Caliper

Caliper is where you turn "does it work?" into a number you can argue with. You collect the outputs that matter, write a rubric your team actually agrees on, and send the work to the people whose judgment counts. The result is a measured baseline you can ship against — and watch over time.

The nouns

Caliper has a handful of things, and almost every workflow is a combination of them.

1. 1
   Datasets

   A dataset is a collection of items in the shape your team actually works with — raw text, chat transcripts, Q&A pairs, file attachments. Items are the rows; a review or eval scores them one at a time. More →

2. 2
   Rubrics

   A rubric is the set of criteria used to score each item. Each criterion has a name, a description, and a numeric scale. Rubrics are reusable — the same rubric can drive many reviews and evals. More →

3. 3
   Reviews

   A review pairs a dataset with a rubric and collects ratings from humans. Launched from inside a dataset's detail page. The rubric is snapshotted at creation, so later edits to the live rubric don't change past results. More →

4. 4
   Evals

   An eval pairs a dataset with a rubric and scores items automatically with an LLM judge. Use evals for ongoing CI-style checks; use reviews when you want human judgment in the loop. More →

A typical loop

A team's first end-to-end run usually looks like this:

1. 1
   Upload a dataset

   Drop in 20–100 representative outputs from the AI feature you're measuring. Don't cherry-pick the good ones; the value of the dataset is that it surfaces the cases you'd rather forget.

2. 2
   Write a rubric

   Three or four criteria, each with a 1–5 scale and a description specific enough that two raters who haven't talked would agree on what a 4 looks like. Vague rubrics are the leading cause of noisy results.

3. 3
   Launch a review

   From inside the dataset, pair it with the rubric. The review owns its own snapshot of the rubric — future edits to the live rubric don't reshape the review.

4. 4
   Send the work

   Invite teammates by email, or generate a share link for guest raters. Guests don't need a workspace seat; they identify themselves on the rating form.

5. 5
   Read the results

   The review overview shows progress, a highlights band (lowest-scored items, biggest rater disagreement), and a per-item breakdown with mean / min / stddev and the rater list. Use it to decide what to fix before the next version ships.

When to reach for Caliper

Common triggers:

• You're about to ship a change to a prompt, model, or retrieval setup and you want to know whether it actually got better.
• You've got an AI feature in production and need a recurring quality signal — weekly check-ins on what users are seeing.
• Stakeholders (legal, product, domain experts) need to weigh in on output before a launch, and you want their feedback structured instead of in a thread.

Where to go next

• Quickstart — dataset → rubric → eval → score, end to end, in about ten minutes
• Run evals in CI — the killer recipe: score your model on every change.
• Datasets — item shapes, upload formats, how to keep them healthy
• Rubrics — designing criteria, scales, descriptions that survive disagreement
• Reviews — pairing dataset + rubric, launching, reading human-rater results
• Evals (API) — LLM-judge auto-scoring from your CI / SDK
• Sharing & invites — bringing in teammates and guest raters