POST/1.0/caliper/evals/:id/runs

Submit a run against an eval. For evals with an external target, the body MUST carry items: [{ itemId, actualOutput }] — Caliper skips inference and goes straight to scoring with the LLM judge.

Required scope: caliper:evals:write (scopes reference).

Returns 202 Accepted immediately with the new run in PENDING status. Scoring proceeds asynchronously on Caliper's side.

Path parameters

idstringRequired

The eval's id (see GET /1.0/caliper/evals/:id).

Request body

triggeredBy"ci" | "ui" | "workbench_publish" | "agent"Required

Informational tag for "what kicked this off." Use "ci" for CI / SDK calls; "agent" for runs an AI agent initiates. Surfaces as the Trigger column on the runs table — analytics breakdown, not access control.

itemsarrayRequired

The actuals your infrastructure produced. Required for evals with an external target (the only kind this endpoint accepts today). Each entry pairs an itemId from GET /1.0/caliper/evals/:id's dataset.items[].id with the string actualOutput your model produced.

items[].itemId — string. Must reference a dataset item that exists in the eval's bound dataset. Unknown ids reject the whole run with 400 validation (the message lists the offenders).
items[].actualOutput — string. The model's output to be scored. The judge compares this against the dataset item's expectedOutput (or expectedResponse for chat items) using the rubric criteria.

Subsets are fine — if a script can't produce an actual for some item (e.g. inference timed out), just omit it. The run will score only the items present; the missing rows don't error.

Posting items against a workbench_flow eval rejects the run with 400 validation (the message says items aren't accepted for flow-target evals) — those runs come from the dashboard, not this API.

labelstring

Optional free-text label for this run. Surfaces in the runs table to help humans recognize which CI build / experiment a run came from. Common pattern: "PR #1234", "main@abc1234", "experiment-tone-v2".

taskOverrideobject

Optional per-run config overrides. The base config is immutable post-create; this lets you tag a single run with variant metadata (e.g. model: "gpt-4o", temperature: 0.2) so cross-run comparisons can group by override. Strict-shape: unknown keys are rejected with a validation error.

Accepted keys: flowStage ("draft" | "published"), flowRevisionId, model, temperature (0–2), systemPrompt. None are interpreted by the judge — they're stored on the run for analytics.

Response (202)

The freshly-created run. Status is PENDING; Caliper transitions it to SCORING and then DONE as the LLM judge processes each item.

idstring

The run's id. Use for polling status.

status"PENDING" | "RUNNING" | "SCORING" | "DONE" | "FAILED"

Current lifecycle state. For external runs, you'll typically see PENDING (or SCORING) on the 202 response, then DONE (or FAILED) once scoring settles.

triggeredBy"ci" | "ui" | "workbench_publish" | "agent"

Echo of the request value.

labelstring | null

Echo of the request value (or null if omitted).

totalItemsinteger

Number of items in this run (= the items[] length the script POSTed).

completedItemsinteger

How many have finished scoring. Increments as the judge processes the queue.

failedItemsinteger

How many failed scoring (judge error, malformed output, etc.). Doesn't fail the whole run unless every item failed.

overallScorenumber | null

Aggregate scalar across items × criteria, normalized to 0–1. null until the run reaches DONE. The eval's latestScore is the most-recent overallScore across all DONE runs.

errorMessagestring | null

Set when status === "FAILED" with a one-line summary. Per-item failures (judge errors, malformed inputs) are on the individual item rows, not here.

createdAtstring

ISO-8601 timestamp.

Example

curl -X POST https://api.zerowidth.ai/1.0/caliper/evals/eval_8a2c91.../runs \
  -H "Authorization: Bearer zw_live_abc12345_..." \
  -H "Content-Type: application/json" \
  -d '{
    "triggeredBy": "ci",
    "label": "PR #1234",
    "items": [
      { "itemId": "item_001", "actualOutput": "We accept refunds within 30 days." },
      { "itemId": "item_002", "actualOutput": "Digital products are non-refundable per our policy." }
    ]
  }'

{
  "id": "run_b41e7d...",
  "evalId": "eval_8a2c91...",
  "status": "PENDING",
  "triggeredBy": "ci",
  "label": "PR #1234",
  "totalItems": 2,
  "completedItems": 0,
  "failedItems": 0,
  "overallScore": null,
  "errorMessage": null,
  "createdAt": "2026-05-20T15:22:08.000Z"
}

Errors

Endpoint-specific error codes on top of the shared error envelope:

Status	`code`	When
`400`	`validation`	The request doesn't fit the eval — the message says which way: the eval has an `external` target but the request omitted `items` (add the array); the eval has a `workbench_flow` target and `items` was posted (those runs come from the UI, not this API); or one or more `itemId`s aren't members of the eval's bound dataset (the message lists the offenders — re-fetch via GET /1.0/caliper/evals/:id, items may have changed since you last cached the list).
`404`	`not_found`	The eval id doesn't exist in the key's workspace.
`409`	`conflict`	The eval was archived. Un-archive from the dashboard before posting new runs.
`403`	`forbidden`	A Personal key was used. Mint a Workspace-wide key with the `ci_evals` preset and retry.

Auth + scope codes (401, 403 scope_missing) come from the shared envelope.

Polling for status

There's no public GET /1.0/caliper/evals/:id/runs/:runId endpoint yet — that's the next addition. Until then, watch the run from the dashboard (caliper.zerowidth.ai/w/<slug>/evals/<evalId>/runs/<runId>) or call GET /1.0/caliper/evals/:id periodically and read latestScore to detect when a run has completed.

Testing your integration

Mint a key
From accounts.zerowidth.ai → Workspace → API keys → New API key. Pick Workspace-wide kind + CI evals preset + leave the cost cap at whatever fits your workload. Copy the raw key when shown — it's the only time you'll see it. (API keys reference)
Create an external eval
From the Caliper dashboard, open Evals → New eval. In the wizard, pick a dataset + rubric, then in the Target step pick External. Give it a label (e.g. "CI"). Save.

Fetch + post

export ZW_API_KEY='zw_live_…'
export ZW_EVAL='eval_…'

# Fetch — note the dataset.items[].id values you'll need
curl https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL \
  -H "Authorization: Bearer $ZW_API_KEY"

# Submit with one fake actual
curl -X POST https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL/runs \
  -H "Authorization: Bearer $ZW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"triggeredBy":"ci","items":[{"itemId":"<one-from-fetch>","actualOutput":"test"}]}'

4
Confirm failure paths
- Tamper with the secret half → 401 auth_invalid.
- Use a Personal key → 403 forbidden.
- Use an eval id from another workspace → 404 not_found.
- Submit an unknown itemId → 400 validation.
- Submit items against a workbench_flow eval → 400 validation.

Troubleshooting

403 forbidden — personal key used

A Personal key was used. These endpoints accept only Workspace-wide keys — mint one with the ci_evals preset and retry.

400 validation — items not accepted

You posted items to an eval whose target is a Workbench flow. Those runs come from the dashboard; this endpoint is for external evals only.

400 validation — unknown item ids

One or more itemIds aren't in the eval's bound dataset — usually a stale cache after items changed. Re-fetch with GET /1.0/caliper/evals/:id and use its dataset.items[].id values.

How do I poll for the score?

There's no public run-poll endpoint yet. Re-fetch the eval with GET /1.0/caliper/evals/:id and read latestScore, or watch the run in the dashboard.

6 min read