Compare runs & read heatmaps

A single eval score tells you where you are. Comparing runs tells you whether you're getting better or worse — and what changed. Caliper gives you two heatmaps over an eval's run history for exactly this.

The overall trend

Start with the overall score across runs. A steady line is a stable feature; a step down right after a prompt or model change is the thing to investigate. But "the number dropped" isn't actionable on its own — that's what the heatmaps are for.

Criterion trend — what got worse

The criterion-trend heatmap is runs × criteria. Read down a column to see how one criterion moved across runs; read across a row to see one run's profile. This is how you catch that a change held tone steady but cost you accuracy — the kind of trade-off an averaged score hides.

Item trend — where it got worse

The item-trend heatmap is items × runs. The same handful of items failing every run is a structural problem — an edge case your flow doesn't handle — and a different thing entirely from scores that jitter randomly run to run. Find the persistently-red rows; they're your worklist.

From a red cell to a fix

Every judge score carries the judge's reasoning. When a cell is red, open it and read why the judge marked it down — then that's a concrete failure to reproduce in Workbench, fix, and re-run. The heatmap going green next run is your proof.