Postil

postil plan: config dry-run

Review configuration is usually tuned by trial and error against live pull requests, for weeks. postil plan replaces that loop with a deterministic preview: apply a candidate config to reviews that already happened and see exactly what would change.

How it works

Every completed review leaves an envelope: the full set of findings the model produced, including ones suppressed by your current thresholds. postil plan re-applies the candidate config's filters (severityThreshold, minConfidence, ignore globs, maxFindings, gate.failOn) to those stored envelopes.

No model calls are made. The plan is exact for filtering changes, free to run, and instant.

Usage

Store envelopes as you review, then evaluate a candidate config against them. A candidate like this (a stricter minConfidence plus an ignore glob for generated code) is a common shape:

# .postil.candidate.yaml
minConfidence: 0.75
ignore:
  - "generated/**"

The command accepts a directory of stored envelopes and reports the before and after finding counts, suppressions, and gate outcome for each one. This transcript was captured from the CLI replaying a public evidence envelope against a candidate config:

$ postil plan --envelopes .cache/docs-plan/envelopes --config .cache/docs-plan/.postil.candidate.yaml

postil plan: replaying 1 stored review(s) under candidate config (.cache/docs-plan/.postil.candidate.yaml)

  swapped-shas-public-evidence.json: 2 -> 0 finding(s); gate: FAILING -> passing
      would suppress: src/app/docs/quickstart/page.tsx:71 [error] Fix cli-ref to use CLI repository SHA
      would suppress: src/app/docs/page.tsx:92 [error] Fix cli-ref to use CLI repository SHA

Summary: 2 finding(s) would be suppressed; 1 gate outcome(s) would change.

Each envelope row identifies the input file, finding count before and after filtering, and gate state before and after. Suppression rows name the affected path, line, severity, and finding title. The summary aggregates suppressed findings and changed gate outcomes.

What it answers

  • "If we raise minConfidence to 0.75, how many of last month's comments disappear, and were any of them ones we acted on?"
  • "If the gate also fails on warn, how many merged PRs would have been blocked?"
  • "Does ignoring generated/** hide anything that ever produced an error-severity finding?"

Limits, stated plainly

postil plan re-filters what the model already said. It cannot predict findings a different reviewer.focus or model would have produced; those change the model call itself. For filter and threshold changes (the overwhelming majority of config churn) the preview is exact; for model changes it tells you so instead of guessing.

Envelope format details are in the envelope schema reference.