Troubleshooting

An eval run has two levels where things can go wrong: the run (did the whole thing execute?) and the items (did each case get an output and a score?). When a run fails, the reason sits on the run. When one case fails, the reason sits on that row. So the explanation is usually one click from the symptom. This page maps the symptoms.

How a run progresses

A run is pending, then running, then scoring, and ends done or failed. Only an eval pointed at a Workbench flow has a running stage — that's where Caliper produces the answers before judging them. Scoring is the judge reading them, which every eval does.

Submitting a run doesn't block. It's created right away and keeps going after that, whether you started it from Caliper or the API, so the status moves on its own while you watch.

The run shows `FAILED`

A run ends without a score in three situations, and the run says which (the third is you pressing Stop):

Setup failed before any item ran — the eval's target no longer resolves. The flow it pointed at was deleted, the pinned version no longer exists, or the target is otherwise invalid. Fix the eval's target (or re-point it) and run again.
Every single item failed. Item failures don't sink a run on their own. When no item produced a scorable result, there's nothing to aggregate, and the run is marked FAILED with a summary. The real diagnosis is at the item level; read on.

Anything less than "all of them" settles as DONE, with the failed items counted separately.

The run is `DONE` but items failed

Each failed item row carries its own errorMessage. Two families:

Inference failures (Workbench-flow targets only) — the flow run behind that item errored, timed out, or the workspace ran out of inference credit partway through the batch. The item's message is the flow's error. Since these are per-item flow runs, the Workbench troubleshooting page applies to them directly.

Judge failures — the item had an output, but scoring it failed: the judge call errored, or its response didn't parse into scores. The item is marked failed with the judge's error rather than being given an invented number.

One thing to keep in mind reading the score afterward: the run's aggregate covers the items that completed. A run where 3 of 40 items failed is scored on 37 — compare against other runs with that in mind, and re-run once the cause is fixed if the failures cluster (three failed edge cases are exactly the ones you care about).

The eval won't run against my flow

Workbench-flow evals have a stage: they run against the flow's published revisions by default — null revision means "latest published at run time," or you pin a specific one. Two ways this bites:

The flow has never been published. There's no published revision to run. Either publish one, or set the eval to the draft stage — the CI-style mode that tests the live working copy before you cut a snapshot.
The pinned revision no longer resolves — it was part of a flow that got re-imported or removed. Re-point the eval at a current revision or at "latest published."

The revision a run actually used is recorded on the run itself, so past runs stay interpretable even after "latest" moves.

I edited the rubric and nothing changed

That's expected. The rubric is copied onto the eval when you create it — editing the live rubric later never reshapes past runs, and it doesn't flow into the eval either. That's what makes run-over-run trends comparable: every run was scored against the same criteria. To score against the revised rubric, create a new eval bound to it — and treat the score series as starting fresh, because the measuring stick changed.

CI submissions get rejected

The submit-run endpoint is strict about what it accepts, and each rejection names the mismatch:

Symptom	Cause	Fix
`400 validation` — the message says `items` is required	The eval's target is `external`, but the request had no `items`.	Send `items: [{ itemId, actualOutput }]` — external evals score what you send.
`400 validation` — the message says items aren't accepted	You posted `items` to a Workbench-flow eval.	Those runs come from the dashboard, where Caliper runs the inference itself. Use an `external` eval for the CI loop.
`400 validation` — the message lists unknown item ids	One or more `itemId`s aren't in the eval's bound dataset — usually a stale cached list after items changed.	Re-fetch GET /1.0/caliper/evals/:id and use its current `dataset.items[].id` values.
`409 conflict`	The eval was archived.	Un-archive it from the dashboard before posting runs.
`403 forbidden`	A Personal key was used.	These endpoints take Workspace-wide keys — mint one with the `ci_evals` preset.

Auth, scope, and rate-limit failures use the shared envelope like everywhere else.

Two semantics worth knowing up front:

Subsets are legal. If your script couldn't produce an actual for some item, omit it — the run scores what arrived and the missing rows don't error. The flip side: the run's score covers only the items you sent, so a shrinking subset can move the number without the model changing.
To detect completion from a script, fetch the run (GET /1.0/caliper/evals/:id/runs/:runId) until its status is terminal, or re-fetch the eval and watch latestScore. The dashboard streams progress live.

Share links are revocable, and a revoked link shows a "this link has been revoked" message in place of the rating form — the most common cause of "the link is broken" reports. Ratings already submitted through a revoked link are kept; only new visits are blocked. Generate a fresh link from the review's Share dialog if the revocation was accidental.

Two other guest-rater behaviors that read as bugs but aren't:

Guests must identify themselves — name + email — before rating. That identity sticks for their session, which is what lets a guest stop mid-review and resume where they left off.
Guest access is scoped to that one review. A guest who asks "why can't I see the rest of the workspace?" is seeing the intended boundary. If someone needs broader access, invite them as a workspace member instead.

ReferenceSubmit a run

The full external-run contract, including every error code.

GuideRun evals in CI

The fetch → run your model → submit → read-the-score loop.

GuideCompare runs

Telling a real regression from variance in the heatmaps.

GuideAPI troubleshooting

Auth, quota, and rate-limit failures common to every endpoint.

6 min read