Evals

An eval is a review with the human replaced by a model. It pairs a dataset with a rubric, points at something that produces outputs, and an LLM judge scores every item against every criterion. The reason to use one over a human review: it runs in seconds, on every change, in CI — so a regression shows up as a falling number instead of a customer complaint.

What's being scored: the target

An eval's target is where the outputs come from. Two kinds:

• Workbench flow — Caliper runs a flow itself on each dataset item, then judges the result. Best for "is my flow good?" You don't need the API for this — run it from the dashboard or the flow's dev-drawer Evals tab.
• External — Caliper runs no inference. You produce outputs in your own infrastructure and submit them via the API. Best for "is my model / pipeline good?", and the path that wires into CI.

A run

Each time an eval runs, you get a run: it moves PENDING → RUNNING → SCORING → DONE (or FAILED), and when it settles each item carries a judge score with the judge's reasoning — so a low number always comes with a why. The rubric is snapshotted onto the eval at creation, so editing the live rubric later never reshapes a past run.

Reading trends across runs

One score is a data point; the value is the trend. Caliper gives you two heatmaps over an eval's run history:

• Criterion trend (runs × criteria) — which criterion moved. Tone held but accuracy slipped after that prompt change.
• Item trend (items × runs) — which cases got harder. The same three edge cases failing every run is a different problem than random noise.

That's how you tell a real regression from variance, and where to look when something drops.

Eval vs. review

Same dataset + rubric pairing; different scorer. Reach for an eval for scale and CI — automatic, repeatable, cheap. Reach for a review when the judgment genuinely needs people. Many teams use both: a fast eval gate on every change, a periodic human review to keep the judge honest.

Where to go next