Evals

An eval is a review with the human replaced by a model. It pairs a dataset with a rubric, points at something that produces outputs, and an LLM judge scores every item against every criterion. The reason to use one over a human review: it runs in seconds, on every change, in CI — so a regression shows up as a falling number instead of a customer complaint.

What's being scored: the target

An eval's target is where the outputs come from. Two kinds:

• Workbench flow — Caliper runs a flow itself on each dataset item, then judges the result. Best for "is my flow good?" You don't need the API for this — run it from Caliper, or from Devtools → Evals inside the flow.
• External — Caliper runs no inference. You produce outputs in your own infrastructure and submit them via the API. Best for "is my model / pipeline good?", and the path that wires into CI.

A run

Each time an eval runs, you get a run: it moves from pending to running to scoring, and ends done or failed. When it settles, each item carries a judge score with the judge's reasoning — so a low number always comes with a why. The rubric is snapshotted onto the eval at creation, so editing the live rubric later never reshapes a past run.

Stop run ends one early. Items already finished keep their results; the ones it cut short say so, and Run the rest picks the run back up from there rather than starting again.

A conversation, not an answer

When an item is one of the multi-turn shapes, the run shows the whole conversation instead of a single output.

A sequence item plays its scripted user turns in order, and the model writes every reply. You read it turn by turn, and it closes with how long it went.

A simulated item is a person Caliper plays. Each turn says what they're doing and why, their message is marked Simulated, and the turn where they got what they wanted is marked in the transcript. It ends with one plain sentence:

• The person got what they were after on turn 4, of 9.
• The person gave up after 6 turns.
• The budget ran out after 12 turns without the person getting there.

That sentence is usually the finding. A conversation the person won on turn two and one they won on turn nine are not the same result, even when both end in success.

Reading trends across runs

One score is a data point; the value is the trend. Caliper gives you two heatmaps over an eval's run history:

• Criterion trend (runs × criteria) — which criterion moved. Tone held but accuracy slipped after that prompt change.
• Item trend (items × runs) — which cases got harder. The same three edge cases failing every run is a different problem than random noise.

That's how you tell a real regression from variance, and where to look when something drops.

Eval vs. review

Same dataset + rubric pairing; different scorer. Reach for an eval for scale and CI — automatic, repeatable, cheap. Reach for a review when the judgment genuinely needs people. Many teams use both: a fast eval gate on every change, a periodic human review to keep the judge honest.

Where to go next