Caliper API reference

Caliper exposes endpoints for the CI / SDK scoring loop: fetch an eval's bound dataset items, run inference against them in your own infrastructure, and POST the actuals back for Caliper to score against the rubric.

Authentication, key management, and the shared error envelope live in API setup.

When to reach for the API

Most Caliper surfaces live in the SPA. The API exists for one specific shape: evals with external targets. The author created an eval bound to a dataset + rubric, but chose "External" as the target so Caliper doesn't run inference itself — that's the script's job.

If you created the eval with a Workbench flow target, you don't need this API. Caliper runs the inference itself when you click Start new run in the UI.

Base URL

https://api.zerowidth.ai/1.0

All endpoints return JSON. All requests require a bearer token. See Authentication for the key format + how to mint one.

For Caliper, mint a Workspace-wide key with the ci_evals preset — that bundles caliper:evals:read + caliper:evals:write in one click. Personal keys aren't accepted on these endpoints yet; the loop is designed for shared CI infrastructure.

Endpoints

The CI loop, end to end

# 1. Fetch the eval — response includes dataset.items[] with id + input
curl https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL \
  -H "Authorization: Bearer $ZW_API_KEY"

# 2. Run inference yourself (whatever model + harness you use)
#    Build a list of { itemId, actualOutput } pairs from the result.

# 3. Submit the actuals — Caliper scores them with the LLM judge
curl -X POST https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL/runs \
  -H "Authorization: Bearer $ZW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "triggeredBy": "ci",
    "label": "PR #1234",
    "items": [
      { "itemId": "item_abc...", "actualOutput": "Paris" }
    ]
  }'

The POST returns a 202 Accepted immediately with the run's id; scoring runs asynchronously on Caliper's side. Poll the run from the dashboard (or via the upcoming GET /1.0/caliper/evals/:id/runs/:runId) to watch it transition from PENDING → SCORING → DONE.