Model bench.

Generated 2026-08-19 with postil 0.8.26. 57 fixtures carry a seeded defect and 13 are clean pull requests where the correct review is silence. Each model saw all 70 once.

Download the raw report (JSON). It carries the fixture-corpus, evaluator and binary digests, so you can tell whether any other table is measuring the same thing.

What the columns mean

Detected is the share of seeded defects the model flagged. Gate correct is how often the merge gate blocked exactly when it should have, which is the number that decides whether a review costs someone a merge. Silent on clean counts the clean pull requests the model left alone. False findings counts findings raised against no defect. No envelope counts runs that never produced valid structured output, which fail closed rather than passing unreviewed code.

Read the column, not the decimal

Each figure is one run of a non-deterministic model. We ran two models four times each to measure how much that matters: detection rate moved over roughly nine percentage points for a single unchanged model, which is wider than the gap between most rows in the table. Treat differences under about five points as noise.

Gate correctness and silence on clean pull requests were far steadier across repeats, and they separate the models that detection rate cannot. If you take one thing from this table, take that: the headline metric is the least useful one on it.

Cost is not the sticker price

Total cost here is what the run actually billed, not a rate card. How much a model reasons before answering moves it far more than its per-token price does. The cheapest model per token in this table emitted more than half a million completion tokens to finish the set; the cheapest model per review emitted thirty thousand and scored higher. Price the work, not the tokens.

Routing changes the answer

The rows above come from screening runs, which let the router pick any endpoint. Hosted reviews cannot do that: they require a zero-retention endpoint, pin one upstream provider, and apply a price ceiling. Re-running two models under that contract moved the numbers enough to change which one we chose.

One model was unaffected; the other lost its entire latency advantage, because the endpoint the router had been choosing for it was not one a zero-retention deployment can use. If you are picking a model for a privacy-constrained deployment, measure it on the route you are allowed to take, not the cheapest one available.

What this cannot tell you

These are seeded defects on synthetic pull requests. A capable model can saturate them, so a high score shows that a model handles the review pipeline and obvious bugs at the stated cost, not that it will find the subtle bug in your codebase. The clean fixtures are the more transferable half: a model that cannot stay quiet here will not stay quiet on your diffs either.

These are our fixtures, and we build the product they score. Read the five-point test for benchmarks and apply it to this page too. The reproduction command below is the honest answer to that problem: run it yourself.

Run it yourself

The harness is in the CLI repository under bench/. Live mode spends real inference tokens and never prints your key.

cargo build --release
cd bench && bun install --frozen-lockfile
export MODEL_API_KEY=...
REVIEW_MODEL=openai/gpt-5.6-luna \
  bun run bench:live -- --json-out .runs/luna.json

One model per run. Compare only against runs whose fixtureCorpusSha256 and evaluatorSha256 match yours; a different corpus is a different test.