Run evals in CI

The point of an eval is to run on every change. This is the loop that wires Caliper into CI: fetch the eval, run your model, submit the outputs, read the score — and fail the build if it dropped.

1. 1
   Create an external eval

   In Caliper, create an eval over a dataset with expectedOutputs and a rubric, and pick External as the target. Note its id.

2. 2
   Fetch the items

   GET /1.0/caliper/evals/:id returns the eval plus its bound dataset items inline — so one call gives your script everything it needs to run inference.

   curl https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL \
     -H "Authorization: Bearer $ZW_API_KEY"

3. 3
   Run your model, submit the actuals

   Produce an output for each item, then POST them back. Use an Idempotency-Key so a CI retry doesn't double-submit (idempotency).

   curl -X POST https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL/runs \
     -H "Authorization: Bearer $ZW_API_KEY" \
     -H "Idempotency-Key: $CI_RUN_ID" \
     -H "Content-Type: application/json" \
     -d '{"triggeredBy":"ci","label":"PR #1234","items":[
           {"itemId":"item_abc…","actualOutput":"…your model output…"}
         ]}'

   It returns 202 Accepted with a PENDING run; the LLM judge scores asynchronously.

4. 4
   Read the score and gate the build

   Scoring is async, and there's no public run-poll endpoint yet — re-fetch the eval with GET /1.0/caliper/evals/:id and read its latestScore, or watch the run in the dashboard. Compare latestScore to your threshold and exit non-zero to fail the build when it drops.

See the full request/response shapes on submit a run and read an eval.