Troubleshooting

An eval run has two levels where things can go wrong: the run (did the whole thing execute?) and the items (did each case get an output and a score?). Caliper keeps the error context at the level where the failure happened — a run-level errorMessage for run-level failures, a per-item message on each failed row — so the diagnosis is usually one click deep. This page maps the symptoms.

The run lifecycle, briefly

A run moves PENDING → RUNNING → SCORING → DONE (or FAILED). RUNNING is the inference phase — only evals with a Workbench flow target have one, because Caliper is executing the flow per item. SCORING is the judge phase, for every eval kind. Runs settle asynchronously: submitting one (from the dashboard or the API) returns immediately with PENDING, and the statuses advance on Caliper's side.

The run shows `FAILED`

A run only fails wholesale in two situations, and the run's errorMessage says which:

Setup failed before any item ran — the eval's configuration no longer resolves. The bound flow was deleted, the pinned revision doesn't exist, the task configuration is invalid. Fix the eval's target (or re-point it) and run again.
Every single item failed. Item failures don't sink a run on their own. When no item produced a scorable result, there's nothing to aggregate, and the run is marked FAILED with a summary. The real diagnosis is at the item level; read on.

Anything less than "all of them" settles as DONE, with the failed items counted separately.

The run is `DONE` but items failed

Each failed item row carries its own errorMessage. Two families:

Inference failures (Workbench-flow targets only) — the flow run behind that item errored, timed out, or the workspace ran out of inference credit partway through the batch. The item's message is the flow's error. Since these are per-item flow runs, the Workbench troubleshooting page applies to them directly.

Judge failures — the item had an output, but scoring it failed: the judge call errored, or its response didn't parse into scores. The item is marked failed with the judge's error rather than being given an invented number.

One thing to keep in mind reading the score afterward: the run's aggregate covers the items that completed. A run where 3 of 40 items failed is scored on 37 — compare against other runs with that in mind, and re-run once the cause is fixed if the failures cluster (three failed edge cases are exactly the ones you care about).

The eval won't run against my flow

Workbench-flow evals have a stage: they run against the flow's published revisions by default — null revision means "latest published at run time," or you pin a specific one. Two ways this bites:

The flow has never been published. There's no published revision to run. Either publish one, or set the eval to the draft stage — the CI-style mode that tests the live working copy before you cut a snapshot.
The pinned revision no longer resolves — it was part of a flow that got re-imported or removed. Re-point the eval at a current revision or at "latest published."

The revision a run actually used is recorded on the run itself, so past runs stay interpretable even after "latest" moves.

I edited the rubric and nothing changed

By design. The rubric is snapshotted onto the eval at creation — editing the live rubric later never reshapes past runs, and it doesn't flow into the eval either. That's what makes run-over-run trends comparable: every run was scored against the same criteria. To score against the revised rubric, create a new eval bound to it — and treat the score series as starting fresh, because the measuring stick changed.

CI submissions get rejected

The submit-run endpoint is strict about what it accepts, and each rejection names the mismatch:

Symptom	Cause	Fix
`400 validation` — the message says `items` is required	The eval's target is `external`, but the request had no `items`.	Send `items: [{ itemId, actualOutput }]` — external evals score what you send.
`400 validation` — the message says items aren't accepted	You posted `items` to a Workbench-flow eval.	Those runs come from the dashboard, where Caliper runs the inference itself. Use an `external` eval for the CI loop.
`400 validation` — the message lists unknown item ids	One or more `itemId`s aren't in the eval's bound dataset — usually a stale cached list after items changed.	Re-fetch GET /1.0/caliper/evals/:id and use its current `dataset.items[].id` values.
`409 conflict`	The eval was archived.	Un-archive it from the dashboard before posting runs.
`403 forbidden`	A Personal key was used.	These endpoints take Workspace-wide keys — mint one with the `ci_evals` preset.

Auth, scope, and rate-limit failures use the shared envelope like everywhere else.

Two semantics worth knowing up front:

Subsets are legal. If your script couldn't produce an actual for some item, omit it — the run scores what arrived and the missing rows don't error. The flip side: the run's score covers only the items you sent, so a shrinking subset can move the number without the model changing.
There's no run-polling endpoint yet. To detect completion from a script, re-fetch the eval and watch latestScore, or open the run in the dashboard.

Share links are revocable, and a revoked link shows a "this link has been revoked" message in place of the rating form — the most common cause of "the link is broken" reports. Ratings already submitted through a revoked link are kept; only new visits are blocked. Generate a fresh link from the review's Share dialog if the revocation was accidental.

Two other guest-rater behaviors that read as bugs but aren't:

Guests must identify themselves — name + email — before rating. That identity sticks for their session, which is what lets a guest stop mid-review and resume where they left off.
Guest access is scoped to that one review. A guest who asks "why can't I see the rest of the workspace?" is seeing the intended boundary. If someone needs broader access, invite them as a workspace member instead.

ReferenceSubmit a run

The full external-run contract, including every error code.

GuideRun evals in CI

The fetch → run your model → submit → read-the-score loop.

GuideCompare runs

Telling a real regression from variance in the heatmaps.

GuideAPI troubleshooting

Auth, quota, and rate-limit failures common to every endpoint.

6 min read