Seven frontier models try to build the same C# workflow engine

A static review of model-generated .NET code, scored with a deliberately fussy 143-point checklist.

Run date: July 8, 2026
Evaluation mode: manual static review
Task: implement a production-quality in-memory C#/.NET 8 workflow execution library
Important caveat: I did not compile or execute the submissions in this review pass.

This is not a broad intelligence benchmark. It is one narrow thing: I asked several reasoning models to write the same small workflow engine, then I read the outputs like I would read a pull request from someone who claims the code is production-ready.

The prompt is intentionally unglamorous. No UI. No database. No clever product context. Just a DAG, async scheduling, cancellation, retry, deterministic reporting, and tests. That is enough. These are exactly the places where code assistants often sound confident while leaving behind race windows, bad terminal states, fake cancellation support, or tests that only prove the happy path.

The short version: Claude Opus 4.8 produced the strongest reviewed answer, GPT-5.5 was very close, and GLM 5.2 was the most convincing quality-per-cost run. Grok and Xiaomi were strong but messy in different ways. MiniMax and DeepSeek were usable, but they exposed more of the usual edge-case debt.

The scoreboard

The scores below use the 143-point diagnostic rubric included later in this document. One passed checklist item equals one point. No fatal gates, no automatic zeros.

A one- or two-point lead here should not be overread. A five-point pattern should. These scores are useful less as a podium and more as a map of where each model’s code starts to fray.

Cost, runtime, and output shape

The economics matter because some answers were not “one clean shot.” Claude burned a full reasoning-only generation before producing the usable continuation. Xiaomi produced an incomplete first generation, then a complete second answer. I count both realities below because the final code and the actual benchmark spend are not the same thing.

A rough cost-efficiency view, using combined cost where a continuation/retry was needed:

This is where the story gets less neat. Claude won the code review and was the worst spend. GLM did not win, but it is the run I would pay closest attention to if I cared about repeated throughput rather than a single trophy answer.

Why this prompt is a useful trap

A workflow engine sounds simple until it has to be correct.

The easy version is: topologically sort jobs and run them. The real version is: validate malformed graphs before execution, track reverse dependents, bound concurrency, avoid executor calls for skipped jobs, count retries exactly, use async delays, preserve cancellation semantics, avoid state corruption, return deterministic summaries, and write tests that do not just restate the demo.

The task catches several common failure modes:

• Fake async: code that uses async syntax but serializes the whole DAG.
• Decorative cancellation: tokens are accepted but not checked at the decision points that matter.
• Broken skip semantics: direct dependents skip, but transitive dependents leak through.
• Retry accounting drift: attempts count retries, logs count something else, summaries count a third thing.
• Nice-looking tests that are wrong: a surprisingly good discriminator.
• Output hygiene failure: a model can reason its way to the right idea and still deliver a messy artifact.

That last one matters. A coding answer is not only an internal solution; it is an artifact someone might paste into a repo. Duplicated half-implementations and broken test blocks are not cosmetic problems.

Original prompt

You are implementing a production-quality C#/.NET 8 workflow execution library.

Build a small but complete in-memory workflow engine that executes jobs with dependency constraints, retry behavior, cancellation, and deterministic execution reporting.

## Requirements

Implement the following public API or an equivalent API with the same behavior:

```csharp
public sealed record WorkflowSpec(IReadOnlyList<JobDefinition> Jobs);

public sealed record JobDefinition(
    string Id,
    IReadOnlyList<string> DependsOn,
    int MaxRetries = 0,
    int RetryDelayMilliseconds = 0
);

public sealed record JobResult(bool Success, string? Message = null);

public interface IJobExecutor
{
    Task<JobResult> ExecuteAsync(string jobId, CancellationToken cancellationToken);
}

public enum JobFinalStatus
{
    Succeeded,
    Failed,
    Skipped,
    Canceled
}

public sealed record JobSummary(
    string JobId,
    JobFinalStatus Status,
    int Attempts,
    string? Message
);

public sealed record WorkflowRunSummary(
    bool Succeeded,
    bool Canceled,
    IReadOnlyList<JobSummary> Jobs,
    IReadOnlyList<string> EventLog
);

public sealed class WorkflowEngine
{
    public Task<WorkflowRunSummary> RunAsync(
        WorkflowSpec spec,
        IJobExecutor executor,
        int maxDegreeOfParallelism,
        CancellationToken cancellationToken);
}
```

## Functional behavior

1. Validate the workflow before execution.

   * Reject duplicate job IDs.
   * Reject missing dependency references.
   * Reject empty, null, or whitespace job IDs.
   * Reject cycles and include useful information about the cycle in the exception message.
   * Reject `maxDegreeOfParallelism <= 0`.
   * Reject negative retry counts or retry delays.

2. Execute jobs according to dependency order.

   * A job may start only after all dependencies have succeeded.
   * Independent jobs may run concurrently.
   * Never run more than `maxDegreeOfParallelism` jobs at once.
   * If a dependency fails, all downstream dependent jobs must be marked `Skipped`.
   * A skipped job must not call the executor.

3. Implement retry behavior.

   * If a job fails, retry it up to `MaxRetries`.
   * `Attempts` should equal the total number of executor calls for that job.
   * A job succeeds if any attempt returns `Success = true`.
   * A job fails only after all attempts are exhausted.
   * If `RetryDelayMilliseconds > 0`, wait before retries using `Task.Delay` and pass the cancellation token.

4. Implement cancellation correctly.

   * Honor cancellation before starting new jobs.
   * Pass the cancellation token to every executor call.
   * If cancellation is requested, do not start additional jobs.
   * Jobs that have not started and cannot run because of cancellation should be marked `Canceled`, unless they are already deterministically skipped due to failed dependencies.
   * The returned summary should set `Canceled = true` if cancellation was requested during execution.
   * Do not swallow unexpected exceptions silently.

5. Produce deterministic summaries.

   * Return one `JobSummary` per job.
   * Sort summaries by job ID using ordinal string comparison.
   * Include a human-readable event log containing meaningful events such as queued, started, retrying, succeeded, failed, skipped, and canceled.
   * The event log does not need to be globally sorted by time, but it must not be corrupted by concurrency.

6. Use idiomatic modern C#.

   * Use `async`/`await` correctly.
   * Avoid blocking calls such as `.Wait()`, `.Result`, `Thread.Sleep`, or busy waiting.
   * Use thread-safe state management.
   * Use clear exception types and messages.
   * Do not use external NuGet packages for the engine implementation.

7. Include tests.
   Provide unit tests or self-contained test examples that cover at least:

   * A successful linear workflow.
   * A successful branching workflow with parallel jobs.
   * Failure causing downstream skips.
   * Retry success.
   * Retry exhaustion.
   * Cycle detection.
   * Missing dependency detection.
   * Duplicate job ID detection.
   * Cancellation behavior.
   * Maximum parallelism enforcement.

## Output format

Return:

1. The complete implementation code.
2. The test code.
3. A brief explanation of important design choices and complexity.
4. Any assumptions you made.

Do not omit edge cases. Do not replace the implementation with pseudocode.

Rubric

The rescoring uses a flat diagnostic checklist. There are no fatal gates and no automatic zeros. Each passed item is worth one point.

Maximum score: 143 points.

Methodology

Each model received the same prompt. Reasoning or maximum-effort mode was enabled where the provider supported it. I exported each conversation as OpenRouter JSON and extracted the model metadata plus the final assistant output.

For every run I recorded:

• model slug and variant slug;
• provider route when visible;
• reported generated tokens;
• tokens per second;
• wall-clock duration;
• reported cost;
• whether reasoning and final output were separate;
• whether the run required continuation or retry;
• whether the delivered answer was clean, duplicated, malformed, or truncated.

Then I reviewed the implementation and tests manually against the 143 checklist items.

This is a static review. I did not compile the code. I did not run the tests. I did not use a hidden test suite. A real production benchmark should do all three.

Full diagnostic score breakdown

Per-model notes

anthropic/claude-4.8-opus-20260528

Score: 141 / 143
Cost: $0.7987815 final continuation / $2.4290145 combined
Tokens: 18,833 final continuation / 84,369 combined
Time: 2m 45.170s final continuation / 15m 44.730s combined
tok/s: 114.022 final continuation / 89.305 combined

Claude wrote the most convincing final implementation. It was not just broad; it was boring in the right places. The validation was defensive, the scheduler was coherent, the tests reached into the uncomfortable cases, and the answer did not confuse “I passed a token around” with actual cancellation semantics.

The catch is ugly: the first generation produced reasoning only. The usable answer came after continuation. If this were scored as strict single-shot completion, Claude’s result would look very different.

What it got right

• Complete public API and strong integration story.
• Null dependency collections normalized to empty arrays, which most models missed.
• Good cycle detection with useful path reporting.
• Safe coordination of scheduler state.
• Broad tests, including cancellation before start, during running work, and during retry delay.
• Internal faults are surfaced rather than quietly disappearing.

Deductions

• Item 45: the scheduler uses scheduled worker tasks rather than the strictest interpretation of a bounded worker queue.
• Item 139: runtime transitions are controlled, but not routed through exactly one explicit transition function.

Verdict

Best answer in the review. Also the least flattering answer from a cost/control perspective because it needed a continuation after a dead first pass.

openai/gpt-5.5-20260423

Score: 139 / 143
Cost: $0.91867545
Tokens: 25,245
Time: 17m 35.268s
tok/s: 23.923

GPT-5.5 produced the strongest single-generation top-tier answer. It was slower than it should have been, but the artifact itself was disciplined: real validation, real tests, careful scheduler state, and good cancellation thinking.

This answer felt less like a dump of code and more like a system design that had been forced through an implementation. It still lost a few protocol-level points, but those were mostly about strict rubric preferences rather than core behavior.

What it got right

• Strong graph validation before execution.
• Careful cancellation tracking, including workflow-level cancellation state.
• Executor calls outside locks.
• Deterministic summaries and useful event log coverage.
• Self-contained tests that cover the required behaviors plus extra validation edges.

Deductions

• Item 16: did not normalize null dependency collections to empty arrays.
• Item 45: uses scheduled job tasks rather than a stricter bounded worker-queue model.
• Item 118: reserved job tasks are effectively fire-and-forget, even though the scheduler design mostly contains the risk.
• Item 139: transitions are disciplined but not centralized through one explicit path.

Verdict

If I cared about one clean answer without asking for continuation, this is the top run. If I cared about latency, this run was painful.

x-ai/grok-4.5-20260708

Score: 136 / 143
Cost: $0.27704754
Tokens: 46,292
Time: 7m 52.625s
tok/s: 97.947

Grok is the messy one. The best implementation inside the answer is strong, but the delivered output contains duplicated and malformed sections. That matters. A benchmark should not give a model full credit for making the reviewer excavate the correct version from a pile of partials.

The scheduler and validation ideas are good. The artifact discipline is not.

What it got right

• Strong core implementation once the final version is isolated.
• Good channel/coordinator-style scheduling.
• Good dependency graph semantics.
• Solid async and exception handling.

Deductions

• Item 16: null dependency collections not normalized.
• Items 57 and 129: cancellation can win too early over later dependency-failure skip precedence.
• Item 98: lacks a good cancellation-during-retry-delay test.
• Items 106 and 107: the delivered answer includes multiple duplicated/malformed implementation blocks and earlier code with obvious syntax/text problems.
• Item 142: duplicate dependencies are not handled consistently.

Verdict

A strong answer trapped inside an undisciplined output. This is exactly why “final artifact quality” belongs in coding benchmarks.

xiaomi/mimo-v2.5-pro-20260422

Score: 136 / 143
Cost: $0.036923730228 final / $0.093789163116 combined
Tokens: 10,104 final / 75,640 combined
Time: 1m 54.539s final / 21m 7.218s combined
tok/s: 88.214 final / 59.690 combined

Xiaomi’s final answer is much better than its run shape. The first generation did not finish; the second was compact, useful, and surprisingly well tested. It is the kind of result that looks excellent if you only inspect the final answer and much less excellent if you count the path taken to get there.

What it got right

• Strong final implementation after retry.
• Excellent tests relative to most models.
• Clean readable structure.
• Correct core dependency semantics.

Deductions

• Item 12: null job entries are not clearly rejected.
• Item 16: null dependency collections are not normalized.
• Item 45: per-job task style rather than a strict bounded worker model.
• Item 60: workflow Canceled flag is inferred from job status rather than directly from cancellation request/observation.
• Item 70: executor exception policy lacks enough type/message clarity under the strict rubric.
• Item 84: event log does not include queued events.
• Item 142: duplicate dependency handling is not consistent.

Verdict

The final answer is strong. The run was not. I would not ignore it, but I would not call it reliable single-shot behavior either.

z-ai/glm-5.2-20260616

Score: 135 / 143
Cost: $0.110626362
Tokens: 25,102
Time: 4m 49.011s
tok/s: 86.855

GLM is the run I keep coming back to. It did not win, but it produced a clean single answer, at a reasonable speed, for a reasonable cost, without making me sort through wreckage.

Its main weakness is narrow but real: strict cancellation atomics. There are small gaps around executor attempt start and retry boundaries. In normal use, the implementation is probably fine. Under adversarial cancellation tests, it loses points.

What it got right

• Clean single implementation.
• Strong validation and cycle detection.
• Good DAG execution model.
• Deterministic reporting.
• No malformed or duplicated output.

Deductions

• Item 16: null dependency collections not normalized.
• Items 50, 51, 54, and 127: cancellation can slip through around first attempt/retry attempt boundaries.
• Item 98: no explicit cancellation-during-retry-delay test.
• Item 139: transitions are not centralized through one explicit path.
• Item 142: duplicate dependencies are not consistently rejected or normalized.

Verdict

Best practical tradeoff in this batch. Not the highest score, but the least annoying high-quality run.

minimax/minimax-m3-20260531

Score: 131 / 143
Cost: not reported / BYOK
Tokens: 54,571
Time: 6m 25.336s
tok/s: 141.619

MiniMax moved fast and covered the main DAG behavior. The core engine is not embarrassing. The problem is that the surrounding engineering discipline is weaker: tests contain issues, messages can be wrong, and internal fault handling is not robust enough for production claims.

What it got right

• Good core scheduler and graph behavior.
• Strong cycle detection.
• Reasonable concurrency handling.
• Fastest reported throughput in this batch.

Deductions

• Item 16: null dependency collections not normalized.
• Items 73, 76, 131, 132, and 133: internal scheduler faults are not surfaced strongly enough; unexpected exceptions could leave state unresolved.
• Item 82: successful jobs with null messages can get misleading fallback messages.
• Items 88, 96, 97, and 98: test suite bugs and weak cancellation coverage.
• Item 142: duplicate dependency handling is not consistent.

Verdict

Fast and competent on the middle of the task. Less convincing at the edges, where production workflow engines tend to hurt you.

deepseek/deepseek-v4-pro-20260423

Score: 130 / 143
Cost: $0.04172094135
Tokens: 47,918
Time: 13m 49.981s
tok/s: 57.734

DeepSeek had a good central idea and a decent scheduler, but too many defensive-engineering items were missing. It reads like an implementation that handles the intended graph but is less prepared for hostile or sloppy input.

The low cost is real. So are the missing null checks.

What it got right

• Good core dependency graph behavior.
• Solid cycle detection.
• Good skip propagation.
• Generally good retry behavior.

Deductions

• Items 9, 10, 11, and 12: weak null validation for spec, executor, jobs collection, and job entries.
• Item 16: null dependencies not normalized.
• Item 50: insufficient immediate cancellation checking before first executor attempt.
• Item 70: executor exceptions are converted to failure without enough type/message detail.
• Item 88: branching parallel test quality is weak.
• Items 118 and 119: fire-and-forget retry scheduling and terminal-state overwrite risk.
• Item 127: possible executor start after cancellation observation.
• Item 139: transitions are not routed through one explicit path.
• Item 142: duplicate dependency handling is not consistent.

Verdict

Cheap and directionally good, but not as production-ready as the top half of the field.

What actually separated the models

The obvious stuff was not the problem. Almost every model could define the records, find a cycle, and run a linear chain. The separation happened in the unglamorous edges.

1. Cancellation was the real discriminator

Lots of answers passed a token into the executor. Fewer checked the token at every decision boundary where a new executor call could begin. The tricky spots were:

• after a job became ready but before it actually started;
• after a semaphore or scheduler slot was acquired;
• before retry attempts;
• during retry delay;
• when cancellation and dependency failure raced.

That last one is especially revealing. If a dependency failed, downstream jobs should be Skipped, not merely Canceled because a token happened to fire nearby. Several implementations had plausible cancellation behavior until this precedence question appeared.

2. “Tests included” did not always mean “tests useful”

A few answers had impressive-looking tests that would not catch the bugs in their own engine. Some tests had bad assertions. Some tested cancellation only before anything interesting happened. Some never tested retry delay cancellation.

The best answers wrote tests that tried to hurt the implementation.

3. Output hygiene matters

Grok is the clearest case. There was a good implementation in the output, but the answer also contained duplicated, malformed, and partially broken material. In a real code review, that is not a footnote. It creates integration risk.

4. Null and duplicate dependency handling exposed discipline

Many models handled missing dependencies but not null dependency collections. Many built counters that would behave badly if a dependency appeared twice. These are not glamorous bugs, but they are exactly the sort of input-shape bugs a reusable library should decide deliberately.

5. Retry backoff was correct but often inefficient

Several high-scoring models kept the scheduler slot occupied during retry delay. This does not violate the hard cap on concurrent executor calls, and it is not a fatal correctness problem. But it is a scalability smell: an independent ready job can wait behind a sleeping retry.

What I would change in the next benchmark

This static review was useful, but the next version should be less forgiving.

1. Compile everything. No more “appears compilable.”
2. Run a hidden .NET 8 test suite. Especially cancellation races and duplicate dependency inputs.
3. Score single-shot and continuation-allowed separately. Claude and Xiaomi show why this distinction matters.
4. Add a runtime stress test. A 10,000-job DAG would expose scheduling choices that static review only hints at.
5. Track output cleanliness as its own metric. Duplicated partial implementations should not be buried inside a generic maintainability score.
6. Separate price-to-quality from absolute quality. Those are different buying decisions.

Limitations

This document is a manual review, not a proof.

• I did not compile or run the generated code.
• I did not run hidden tests.
• Some scoring calls involve judgement, especially around cancellation semantics and scheduler lifecycle.
• OpenRouter metadata is run-specific: route, provider, region, load, pricing, and throughput can all change.
• Reasoning tokens and final-output tokens were not always separately reported.
• Multi-generation runs complicate fairness. A final answer can be excellent even if the first attempt failed to produce a usable artifact.

Still, static review is not useless. A lot of serious implementation problems are visible without running anything. The important thing is to say what kind of evidence this is and not pretend it is more complete than it is.

Reproducibility notes

The source artifacts were OpenRouter JSON exports from July 8, 2026. The reviewed model slugs were:

• minimax/minimax-m3-20260531
• x-ai/grok-4.5-20260708
• z-ai/glm-5.2-20260616
• deepseek/deepseek-v4-pro-20260423
• xiaomi/mimo-v2.5-pro-20260422
• anthropic/claude-4.8-opus-20260528
• openai/gpt-5.5-20260423

For a stronger public release, I would attach:

• the raw JSON exports;
• extracted final assistant outputs;
• SHA-256 hashes for each extracted output;
• exact provider routing metadata;
• the pass/fail sheet for all 143 items;
• a .NET 8 hidden test harness.

Bottom line

Claude won the static review. GPT-5.5 was the best clean single-generation top-tier run. GLM 5.2 was the result I would most want to rerun at scale. Grok showed why artifact quality matters. Xiaomi showed why final-answer scoring and run-level scoring are not the same thing. MiniMax and DeepSeek were useful but less convincing at the edges.

That is the main lesson here: the models are not failing at the shape of the solution anymore. They are failing, when they fail, at engineering judgment under pressure — cancellation races, malformed output, state transitions, tests that should have been meaner, and the small defensive choices that turn a plausible library into a reusable one.