Caliper

Caliper is where you turn "does it work?" into a number you can argue with. You collect the outputs that matter, write a rubric your team actually agrees on, and send the work to the people whose judgment counts. The result is a measured baseline you can ship against — and watch over time.

The nouns

Caliper has a handful of things, and almost every workflow is a combination of them.

1. 1
   Datasets

   A dataset is a collection of items in the shape your team actually works with — Q&A pairs, chat transcripts, whole conversations the model has to carry, and raw payloads for anything else. Items are the rows; a review or eval scores them one at a time. More →

2. 2
   Rubrics

   A rubric is the set of criteria used to score each item. Each criterion has a name, a description, and a numeric scale. Rubrics are reusable — the same rubric can drive many reviews and evals. More →

3. 3
   Reviews

   A review pairs a dataset with a rubric and collects ratings from humans. Start one from the Reviews tab, or from the dataset or rubric you want it to use. The rubric is snapshotted at creation, so later edits to the live rubric don't change past results. More →

4. 4
   Evals

   An eval pairs a dataset with a rubric and scores items automatically with an LLM judge. Use evals for ongoing CI-style checks; use reviews when you want human judgment in the loop. More →

5. 5
   Specs

   A spec runs the other way round: when you don't yet have the right answer, it gathers what one should be from the people who'd know, finds where they agree, and turns that into the answer an eval can score against. More →

A typical loop

A team's first end-to-end run usually looks like this:

1. 1
   Upload a dataset

   Drop in 20–100 representative outputs from the AI feature you're measuring. Don't cherry-pick the good ones; the value of the dataset is that it surfaces the cases you'd rather forget.

2. 2
   Write a rubric

   Three or four criteria, each with a 1–5 scale and a description specific enough that two raters who haven't talked would agree on what a 4 looks like. Vague rubrics are the leading cause of noisy results.

3. 3
   Launch a review

   From inside the dataset, pair it with the rubric. The review owns its own snapshot of the rubric — future edits to the live rubric don't reshape the review.

4. 4
   Send the work

   Invite teammates by email, or generate a share link for guest raters. Guests don't need a workspace seat; they identify themselves on the rating form.

5. 5
   Read the results

   The review overview shows progress, a highlights band (lowest-scored items, biggest rater disagreement), and a per-item breakdown with mean / min / stddev and the rater list. Use it to decide what to fix before the next version ships.

When to reach for Caliper

Common triggers:

• You're about to ship a change to a prompt, model, or retrieval setup and you want to know whether it actually got better.
• You've got an AI feature in production and need a recurring quality signal — weekly check-ins on what users are seeing.
• Stakeholders (legal, product, domain experts) need to weigh in on output before a launch, and you want their feedback structured instead of in a thread.

Where to go next

• Quickstart — dataset → rubric → eval → score, end to end, in about ten minutes
• Run evals in CI — the killer recipe: score your model on every change.
• Datasets — item shapes, upload formats, how to keep them healthy
• Rubrics — designing criteria, scales, descriptions that survive disagreement
• Reviews — pairing dataset + rubric, launching, reading human-rater results
• Evals (API) — LLM-judge auto-scoring from your CI / SDK
• Sharing & invites — bringing in teammates and guest raters