Generate a dataset with AI
The slowest part of evaluation is usually building the dataset. Caliper can draft one for you from a short brief, so you start with something real to refine instead of a blank table.
From a brief
Describe what you're testing and how many cases you want, and choose how much of each item to fill:
- Inputs only — scenarios now, answers later (heading toward a spec).
- With golden answers —
expectedOutputs, ready to feed an eval. Pick verbatim or success-conditions style depending on whether there's one right answer or many. - With sample answers — captured
outputs, ready for a human review.
Conversations, not just messages
Pick the item shape before you generate:
- Single messages — one input per item, with the answer slots above.
- Scripted conversations — two to six user turns per item that build on each other. Your assistant writes every reply itself, and each item says what a good final reply and a good conversation look like.
- Simulated people — a person Caliper plays against your assistant, adapting to each reply: who they are, what they want, and how they behave. Pick one way of behaving for the whole batch, from a real customer with a real need to someone applying pressure, or let it vary from person to person.
Stress-test with adversarial presets
A test set of friendly cases tells you little about the failure modes that matter. Generation presets bias the cases toward the hard stuff — prompt-injection, jailbreak, and multilingual inputs — so the dataset probes where things actually break, not just the happy path.
For a specific flow
Point generation at a Workbench flow and Caliper uses the flow's own shape to write cases tailored to it — a test set for this flow. It's the fastest way to go from "I built a flow" to "I'm measuring it."
Then refine
Generated items are a starting point. Read them, fix the ones that are off, drop the ones that don't fit, and add the real cases only you know about. The generator gets you to a working draft; your judgment makes it trustworthy.