Generate a dataset with AI

The slowest part of evaluation is usually building the dataset. Caliper can draft one for you from a short brief, so you start with something real to refine instead of a blank table.

From a brief

Describe what you're testing and how many cases you want, and choose how much of each item to fill:

• Inputs only — scenarios now, answers later (heading toward a spec).
• With golden answers — expectedOutputs, ready to feed an eval. Pick verbatim or success-conditions style depending on whether there's one right answer or many.
• With sample answers — captured outputs, ready for a human review.

Stress-test with adversarial presets

A test set of friendly cases tells you little about the failure modes that matter. Generation presets bias the cases toward the hard stuff — prompt-injection, jailbreak, and multilingual inputs — so the dataset probes where things actually break, not just the happy path.

For a specific flow

Point generation at a Workbench flow and Caliper uses the flow's own shape to write cases tailored to it — a test set for this flow. It's the fastest way to go from "I built a flow" to "I'm measuring it."

Then refine

Generated items are a starting point. Read them, fix the ones that are off, drop the ones that don't fit, and add the real cases only you know about. The generator gets you to a working draft; your judgment makes it trustworthy.