The least useful number on our own benchmark

We publish a table of model benchmark results. It was measured against a fixture set that no longer exists, on a build seven minor versions old, and it flattered every small model on it. So we re-ran everything: 19 models, one corpus, one binary, one afternoon. The results changed which model we run, and they changed what we think the table is for.

Detection rate barely separates anything

The column everyone leads with is how many seeded defects a model finds. We ran three models four times each, against the same fixtures with the same binary, to see how stable that number is.

One unchanged model, one unchanged fixture set, nine points of movement. On our full table, 11 of 19 models sit within 9 points of the best one. The headline metric cannot tell most of them apart, and any single-run comparison between two of them is measuring the weather.

This is not a claim that detection does not matter. It is a claim about resolution: if your instrument moves nine points on its own, you cannot report a three-point difference as a result. We had been doing exactly that, including in our release gate, which blocked five of seven releases by comparing one run against a baseline recorded from a single lucky run at the top of the same distribution.

Silence separates them cleanly

Two columns were steady across repeats: how often a model stays quiet on a clean pull request, and how often the merge gate blocks exactly when it should. Those are also the two that decide whether anyone keeps the tool switched on.

Those first three are within five points of each other on detection and nowhere near each other in practice. Gemma 3 interrupts nine of thirteen clean pull requests. Our own published table had it at 97% with one false positive, because the old corpus had fewer adversarial clean cases. Mistral Small was published at 100% detection and zero false positives; on the current fixtures it raises seven false findings and stays quiet on six of thirteen clean diffs.

We had been recommending both. That is the cost of leaving a benchmark table up after the thing it measured has changed underneath, and it is why every figure on the new page carries the digest of the corpus it was scored against.

A version number is not an upgrade

The day before we ran this, the vendor of the model we were using released its successor. Same family, same context window, same price. It failed to produce valid structured output on eighteen of seventy cases, emitted twice the reasoning tokens, and cost three times as much. Upgrading on the version number alone, which is the obvious move, would have been a straight downgrade with a bill attached.

Cost is a reasoning budget, not a price

The cheapest model per token in our sweep charges $0.08 per million input tokens. It emitted 582,000 completion tokens to finish the fixture set and took 87 seconds at the ninety-fifth percentile. The model that won the sweep emitted 31,500 completion tokens, finished in 21 seconds, and cost less in total despite a higher per-token price.

Per-token pricing tells you almost nothing about what a review costs. How much a model thinks before answering tells you nearly everything, and it is not on any pricing page. You have to run it.

We were measuring a route we cannot use

The worst mistake of the exercise was ours, and we made it three times. Our screening harness lets the router pick any endpoint. Hosted reviews cannot: they require a zero-retention endpoint, pin one upstream provider, and cap the price. Every number above was measured without those constraints, and we kept reading them as though they described what customers would get.

Re-running the two finalists under the real contract settled it. One was unaffected. The other, which we had been about to select on the strength of a rock-steady 18-second tail latency, went to 58 seconds once pinned to a zero-retention provider: the endpoint the router had been quietly choosing for it was one we are not allowed to use.

If you run a benchmark to pick a model for a constrained deployment, constrain the benchmark the same way. Otherwise you are measuring a product you cannot ship.

What we changed

Hosted reviews now run on openai/gpt-5.6-luna, pinned to a zero-retention endpoint, replacing z-ai/glm-5.2. It decides the merge gate correctly 86% of the time against the previous model's 74%, raised no false finding in any run, stayed silent on every clean fixture, and costs about a seventh as much. The changelog records the release.

It was a model we had already rejected. In July a preflight guard refused it because projected qualification spend exceeded its cap, so it made zero calls. The price has since fallen fivefold. The rejection was filed in the same shape as a quality failure, so nobody revisited it. A rejection on cost is perishable evidence and ours was not marked as such.

We also re-recorded our release baseline from the median of four runs rather than a single run, which is what stops the gate failing by construction on a metric that moves nine points on its own.

Read it yourself

Every model we scored is on the model bench page, including the ones that did badly and the one that could not be reached at all, with the raw report and the command to reproduce it. The harness is in the CLI repository, and the model catalogue lists what each option costs. These are our fixtures and we build the product they score, so apply our own five-point test for benchmarks to them. The reproduction command is the part that matters.