Caliper quickstart

You'll turn "does this look good?" into a number. By the end you'll have a dataset, a rubric, and a score — and you don't need to arrive with any of it ready. We'll get the data in from scratch.

1. 1
   Open Caliper

   Go to caliper.zerowidth.ai, sign in, and pick your workspace.

2. 2
   Create a dataset and get items into it

   A dataset is the set of cases you measure against. Create an empty one, then fill it whichever way fits — you don't need a file ready:

   • Upload a file — in the dataset, choose Add items → upload, and download a template first (JSON covers every item shape; CSV is the simple Q&A grid). Fill in your cases and drop the file back in.
   • Generate items — let Caliper draft items for you from a short brief; it can produce inputs alone, inputs with golden answers, or inputs with sample answers (more on that next).
   • Add by hand — type a few items directly. Five real cases beat fifty sloppy ones.
3. 3
   Pick the item shape — it decides what you can do

   This is the part that trips people up. What each item carries determines whether you can run an eval, a review, or neither yet:

   An item with…	Use it for	Why
   input + expectedOutput (a golden answer)	Eval	The LLM judge scores a model's actual output against the golden answer.
   input + output (an answer you already have)	Review	People rate the captured answer as-is against your rubric.
   input only (no answer)	Spec	Scenarios now; add expected answers later. Not scorable until you do.

   Two more shapes for non-Q&A data: chat items carry a message transcript (plus an optional expectedResponse), and raw items hold any JSON payload. The downloaded template has one example of each — copy the line that matches your data and delete the rest.

4. 4
   Write a rubric (or start from a preset)

   A rubric is what "good" means, made explicit. The fastest path is to pick a preset — general helpfulness, customer support, code review, safety — and tweak it. Or build your own: add criteria, each with a name and a scale, e.g. Accuracy (1–5) and Tone (1–5). Write the descriptions like you're briefing a careful reviewer; vague criteria are the top cause of noisy scores.

5. 5
   Score it

   • If your items have golden answers, create an eval: pick the dataset + rubric, point it at a Workbench flow (Caliper runs it) or External (you submit outputs via the API), and run. An LLM judge scores every output against every criterion.
   • If your items have captured answers, launch a review instead and send it to the people whose judgment counts.
6. 6
   Read the result

   You get an overall score plus a per-item, per-criterion heatmap — so you see not just that it scored 0.78 but where it's weak. That's the signal you act on.

Make it a habit

A score you check once is a demo; a score that runs on every change is a guardrail. Wire an External eval into CI — fetch the eval, run your model, submit the outputs, read the score back — so a regression fails a pull request instead of reaching a customer.

# Submit your model's outputs for an external eval (the CI shape).
curl -X POST https://api.zerowidth.ai/1.0/caliper/evals/$ZW_EVAL/runs \
  -H "Authorization: Bearer $ZW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"triggeredBy":"ci","items":[
        {"itemId":"<id-from-the-eval>","actualOutput":"…your model output…"}
      ]}'

What's next