Datasets
A dataset is a collection of items in the shape your team actually works with. Items are the rows of a review, eval, or spec — one row per thing that gets looked at. Datasets are workspace-scoped with a visibility setting (Private / Workspace / Shared).
Item kinds
There are five:
- Q&A — an
input(a question or prompt), plus optionally an answer. The most common shape, and the one with the slot model below. - Chat — a fixed message transcript. Every turn, including the assistant's, is written into the item; the model under test only writes the final reply. The shape for rating a captured conversation, and for the one attack where a fabricated earlier assistant turn is the point. May carry an
expectedResponse. - Sequence — the user's turns only, in order. The model under test writes every assistant turn, one at a time, with its own earlier replies in front of it. Carries an
expectedResponsefor the final reply and an optionalexpectedBehaviorfor the whole conversation. This is how the slow attacks get tested: the ones that build up over several messages. - Simulated — a
goal, who the person is, how they behave (adisposition), an optional approach, and a turn budget. Caliper plays that person and adapts to each reply until the goal lands, they give up, or the budget runs out. The run records the outcome and the turn it happened on. The disposition can be a real customer with a real need, someone applying pressure to get something they shouldn't, someone confused, impatient, or vague, or anything you write yourself. Runs only when Caliper runs the eval itself. - Raw — an opaque JSON payload, for anything that doesn't fit the others.
The three Q&A slots
This is the part that decides what you can do with a dataset. A Q&A item always has an input; what else it carries determines whether it feeds a spec, a review, or an eval:
| A Q&A item with… | Feeds a… | Because |
|---|---|---|
input only | Spec | There's no answer yet — you're gathering what the ideal one should be. |
input + output (an answer you already have) | Review | People rate the captured answer as-is against a rubric. |
input + expectedOutput (the golden answer) | Eval | An LLM judge scores a model's actual output against the golden one. |
So a dataset isn't tied to one use — add expectedOutputs and it's eval-ready; collect outputs from production and it's review-ready; leave them blank and it's a spec waiting for answers.
Getting items in
You don't need a file ready to start:
Drop a JSON or CSV file into the upload dialog. Download a template from inside it first — JSON covers every kind and shows all three Q&A slots; CSV is the simple Q&A grid. Fill what you have.
Context variables
Any item except a raw one can carry context variables: name-and-value pairs pinned to that item, filled in under the item's content when you write or edit it.
They're for the runtime context a flow would otherwise pick up fresh each time — current_time = 07:30, a plan tier, a locale. Pinning them per item means a case that depends on "it's Monday morning" tests the same way in March as it did in January. The names have to match the variables the flow expects.
Once set, they're listed under the item wherever you look at it, including inside a run.
How a dataset is used
A dataset can drive many things over its life — the same items scored against different rubrics, by humans in a review, by an LLM judge in an eval, or gathered into golden answers in a spec. Each keeps its own results; deleting one doesn't touch the dataset.
Datasets can also fill themselves from production: a flow's runs can be bookmarked straight into an auto-managed feedback dataset, so your test set grows from real traffic instead of made-up cases.
Editing & deleting
The Edit action changes name, description, and visibility; item content is edited per-item. Deleting a dataset removes it from your list. Reviews and evals that used it stay readable, so past results don't vanish.
Starters
When you want to check an assistant for something everyone worries about and you have no cases of your own yet, start from a starter. Caliper ships five: prompt injection, system prompt leaking, over-refusal, personal data handling, and bias under ambiguity. Each installs a small dataset and a matching rubric into your workspace as copies you own and can edit. The items mix single messages with multi-turn conversations, because the attacks that work are the slow ones: a false memory of what the assistant "already said", a fabricated earlier turn where it seemed to agree, a story that turns into a request on the fifth message. Pick one from the Starters tab of the new-dataset dialog, or ask zv1 to install it.
A starter is a smoke test. Its items are public and original, so a clean run tells you your setup is wired correctly and the obvious cases pass. It does not tell you the assistant is safe. Add your own cases to the copy, since those are the ones that matter.
Where to go next
- Rubrics — what "good" means; the other half of a review or eval.
- Reviews — human multi-rater scoring.
- Evals — automatic LLM-judge scoring.
- Specs — gather the ideal answers an eval scores against.