Run evals in CI

This guide is for a model that does not run on ZeroWidth: your own prompt, your own provider, your own pipeline. Caliper holds the dataset and the rubric, judges every answer, and keeps the run history. Your CI job produces the answers and fails when the score falls under a threshold.

By the end you have an eval in Caliper, a workspace API key, and a job that runs on every pull request. The whole loop is four calls to the public API.

Set up Caliper, once

Create a dataset
The questions your model has to answer, each with a reference answer. Import them as JSON through the upload dialog, or ask zv1 to create the dataset and add the items. Real cases beat invented ones, and eight honest ones are enough to start. See datasets.
Create a rubric
Three or four criteria, each a question the judge can answer from the text, on a numeric scale. Write them before you look at any outputs. See rubrics.
Create an external eval
Create an eval over that dataset and rubric and choose External as the target. Caliper runs no inference for an external eval; it waits for outputs. Copy the eval's id from the URL. zv1 can create this too: ask for an external eval on the dataset and rubric and it hands back the id.
Create a workspace API key
In your workspace settings, create a workspace-wide key with the ci_evals preset. It carries caliper:evals:read and caliper:evals:write. Personal keys aren't accepted on these endpoints. zv1 can't mint keys; this step is yours.

The loop

Your script runs these four calls. Base URL is https://api.zerowidth.ai/1.0, and every call sends Authorization: Bearer <key>.

1
Fetch the eval
GET /1.0/caliper/evals/:id returns the eval with its dataset items inline, so one call gives the script everything it needs.
```
curl https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL \
  -H "Authorization: Bearer $ZW_API_KEY"
```
Each item in dataset.items[] has an id and an input (or messages for chat items, or turns for sequence items). A sequence is a scripted conversation: send the first turn, keep the reply, send the second turn with the reply in front of it, and so on, then submit every reply.
Answer every item
Run your model on each input. Keep the pairs as { itemId, actualOutput }. If an inference call fails, submit the failure text as the answer so the judge scores it, rather than dropping the item and quietly improving the average.

Submit the run

POST /1.0/caliper/evals/:id/runs with the pairs. Send an Idempotency-Key built from your CI run id so a retried job doesn't create a second run.

curl -X POST https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL/runs \
  -H "Authorization: Bearer $ZW_API_KEY" \
  -H "Idempotency-Key: gh-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" \
  -H "Content-Type: application/json" \
  -d '{"triggeredBy":"ci","label":"PR #1234","items":[
        {"itemId":"item_abc…","actualOutput":"…your model output…"}
      ]}'

The response is 202 Accepted with the run in PENDING. Keep its id.

4
Poll the run and gate the build
GET /1.0/caliper/evals/:id/runs/:runId every few seconds until status is DONE or FAILED. Then compare overallScore (0 to 1) to your threshold and exit non-zero when it's under. The response carries every item's per-criterion scores and the judge's reasoning, so the job log can name the criterion that slipped.
```
curl https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL/runs/$ZW_RUN \
  -H "Authorization: Bearer $ZW_API_KEY"
```
Gate on the run you submitted, never on the eval's latestScore. Two overlapping runs make the latest score ambiguous.

The workflow

A GitHub Actions job that runs the script on pull requests touching the prompt, and nightly against main so a change on the provider's side shows up without a commit.

name: Eval
on:
  pull_request:
    paths: ["prompt.md", "bot.mjs"]
  schedule:
    - cron: "0 6 * * *"
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: node eval.mjs --label "PR #${{ github.event.pull_request.number }}" --threshold 0.8
        env:
          ZEROWIDTH_API_KEY: ${{ secrets.ZEROWIDTH_API_KEY }}
          ZEROWIDTH_EVAL_ID: ${{ secrets.ZEROWIDTH_EVAL_ID }}
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}

Label runs with the pull request number. That label is what you'll pick out in Caliper when you compare two runs, and what zv1 cites when you ask how a run went.

Reading the results

In the job log, print the per-criterion means and the lowest-scoring items with the judge's reasoning. A single overall number hides which criterion moved.
In Caliper, open the eval and compare the pull request's run against the last run on main. The cross-run view shows each criterion side by side. See compare runs.
Regression alerts fire for these runs the same as any other. Set the eval's threshold and the eval's creator hears about a drop by notification or email.
Ask zv1. "Why did the score drop on the last run" reads the run, names the criterion that slipped, and quotes the judge on the worst items.

Choosing a threshold

Run the loop a few times on main first and look at the spread. Judges are consistent but not identical run to run, so set the threshold a little under the typical score rather than at it. A stricter threshold on main than on pull requests is common.

5 min read