Postil

Blog

Evidence has to link back

· Postil team

Evidence for an AI code reviewer should be inspectable, not just persuasive. A screenshot or polished demo can show what a product looks like, but it cannot answer the practical question: what code was reviewed, at what state, and where is the artifact that produced the claim? Postil's public evidence page is built around real review output from public Postil repositories, so the examples can link back to source.

The pull request is not a fixed object

A pull request changes over time. New commits arrive, force-pushes replace history, conversations move, and the final merge state may not be the same state a reviewer saw. That is why a useful evidence link has to point at the reviewed commit, not only at the PR conversation.

The work merged in postil-dev/postil#321 added that breadcrumb to the evidence UI. Each evidence card exposes a repository link and a pull-request-files link scoped to the reviewed commit SHA. PR #321 is merged into main; its merge commit is 1b9182c262e55609a60a4723b2c5bd1f74651c1b.

The data keeps the verification record

The public evidence data is in src/data/evidence/index.ts. The file documents the rule it follows: public examples come from Postil's public repositories; finding titles and bodies are copied from check-run annotations; review and gate titles and summaries are copied from their check-runs; check-run IDs and the reviewed head SHA are retained as the verification record.

The page does not rely on invented sample PRs for those cases. Its UI shows the reviewed diff excerpt or complete diff, the finding text Postil produced, and the gate outcome for the reviewed commit. For silent reviews, the evidence case links a pull request where Postil left no visible review comment.

Truthful links are part of the claim

Evidence links also keep the gate and review boundary honest. If a case says the gate failed, the source data has a gate check-run URL and a gate.failing value. If a case says the review advised without blocking, the findings live in the advisory review output while the gate remains passing. If a case says the review was silent, the public PR has no visible Postil review comment.

Some GitHub check-run pages can become unavailable in the UI after GitHub's retention window. Postil still keeps the reviewed head SHA and check-run URLs in the evidence data, while the visible page links to the repository and the PR files at the reviewed commit. That is the durable part of the public claim: readers can inspect the code state the example refers to instead of trusting a fictional demo.