Rubrics
A rubric is the set of criteria a rater uses to score each item. Every criterion has a name, a description, and a numeric scale. Together they're the agreement your team makes ahead of time about what "good" looks like.
Start from a preset
You don't have to design one from scratch. Caliper ships a set of preset rubrics — pick one when you create a rubric and it fills in the criteria for you, ready to use as-is or edit:
- Quick reactions — a single pass/fail criterion, for a fast gut-check.
- General helpfulness — accuracy, helpfulness, tone. A sane default for any text assistant.
- Customer support quality — solves the problem, empathy, accuracy, brand fit.
- Code review quality — correctness of the analysis, severity calibration, explanation quality.
- Safety & calibration — refusal accuracy, confidence calibration, policy adherence.
Presets are starting points. Rename criteria, change the scales, add or drop whatever you need. The rest of this page is for when you want to go beyond them.
Anatomy of a rubric
- Name + description
The rubric itself has a short name and an optional description that explains the scope — what this rubric is meant to evaluate, and what it deliberately isn't.
- Criteria
Each criterion is a separate question raters answer on every item. Three or four criteria is usually right; more than five and rater fatigue eats into quality fast.
- Scale
Per criterion, you set a min and max (Caliper v0 supports numeric scales only — categorical / boolean scales are on the roadmap). 1–5 is a defensible default; 0–1 works for "did the thing pass at all"; 1–10 invites false precision and we don't recommend it.
- Criterion description
The piece that makes or breaks consistency. The description should tell a rater what a 4 looks like vs a 3, and what counts as a 1. Vague descriptions ("Quality") give you noisy ratings; descriptions with concrete anchors ("4 = answers the question fully and grounds it in the right document; 2 = answers the question but cites the wrong source; 1 = hallucinates") give you signal.
Reusability
Rubrics are reusable on purpose. You typically only have a handful of rubrics per workspace ("Customer-support reply quality," "Summarization fidelity," "Retrieval-grounding strength") and each one drives many reviews + evals.
Editing a rubric while a review (or eval) is in progress is safe: when one is created, Caliper snapshots the rubric onto it. Future edits to the live rubric don't reshape past results — those ratings stay attached to the criteria they were submitted against.
Designing criteria
A few patterns worth borrowing:
- One axis per criterion. "Accuracy and clarity" is two criteria; rate them separately so a precise-but-confusing answer doesn't get an averaged-out 3.
- Define the floor. What's a 1? Without a defined floor, raters anchor on different bottoms — your 2 is someone else's 1.
- Avoid the temptation of binary criteria on a 1–5 scale. If a criterion is really yes/no ("Did the answer cite a source?"), say so and rate it that way — don't pretend it's a continuum.
- Reserve a "safety" criterion. If outputs could carry risk (PII, harmful content, hallucination), make a separate criterion for it. Folded into a generic "quality" score, safety failures get averaged away.
Visibility
Same Private / Workspace / Shared setting as datasets. A workspace-visible rubric is the usual choice — you typically want teammates to be able to reuse it on their own reviews + evals.
Where to go next
- Reviews — pair a rubric with a dataset and collect human ratings.
- Evals (API) — same pairing, scored automatically by an LLM judge.
- Datasets — the other half of a review or eval.