Datasets
A dataset is a collection of items in the shape your team actually works with. Items are the rows of a review, eval, or spec — one row per thing that gets looked at. Datasets are workspace-scoped with a visibility setting (Private / Workspace / Shared).
Item kinds
There are three:
- Q&A — an
input(a question or prompt), plus optionally an answer. The most common shape, and the one with the slot model below. - Chat — a turn-by-turn message transcript, for evaluating multi-turn conversations. May carry an
expectedResponse. - Raw — an opaque JSON payload, for anything that doesn't fit the other two.
The three Q&A slots
This is the part that decides what you can do with a dataset. A Q&A item always has an input; what else it carries determines whether it feeds a spec, a review, or an eval:
| A Q&A item with… | Feeds a… | Because |
|---|---|---|
input only | Spec | There's no answer yet — you're gathering what the ideal one should be. |
input + output (an answer you already have) | Review | People rate the captured answer as-is against a rubric. |
input + expectedOutput (the golden answer) | Eval | An LLM judge scores a model's actual output against the golden one. |
So a dataset isn't tied to one use — add expectedOutputs and it's eval-ready; collect outputs from production and it's review-ready; leave them blank and it's a spec waiting for answers.
Getting items in
You don't need a file ready to start:
How a dataset is used
A dataset can drive many things over its life — the same items scored against different rubrics, by humans in a review, by an LLM judge in an eval, or gathered into golden answers in a spec. Each keeps its own results; deleting one doesn't touch the dataset.
Datasets can also fill themselves from production: a flow's runs can be bookmarked straight into an auto-managed feedback dataset, so your test set grows from real traffic instead of made-up cases.
Editing & deleting
The Edit action changes name, description, and visibility; item content is edited per-item. Deleting a dataset is reversible — it disappears from your list, but reviews and evals that reference it stay readable, so historical results don't vanish.
Where to go next
- Rubrics — what "good" means; the other half of a review or eval.
- Reviews — human multi-rater scoring.
- Evals — automatic LLM-judge scoring.
- Specs — gather the ideal answers an eval scores against.