Specs
An eval scores outputs against a known-good answer. But often you don't have the known-good answer yet — that's the hard part. A spec is how you get it: instead of scoring what the AI produced, you gather what a correct answer should be from the people who'd know, find consensus, and turn that into the golden answer an eval can then score against.
How it works
- Pick the cases
A spec runs over a dataset of
input-only Q&A items — the questions you want ideal answers for. - Gather responses
Contributors write their own ideal answer for each item, in plain language. Invite teammates, or send a share link so domain experts can contribute without an account.
- Find consensus and promote
Compare the responses side by side. When the right answer is clear, promote it — it becomes that item's
expectedOutput. The dataset is now eval-ready.
A spec is IN_PROGRESS while you collect, CLOSED once you've promoted what you need.
When to reach for one
- You're standing up a new eval and have questions but no answer key.
- The "right" answer is a judgment call that belongs to experts: support leads, lawyers, clinicians.
- You want the answer key to reflect what several people agree on.
Spec presets give you a starting frame for common shapes — support responses, code review, policy interpretation, tone & voice, advisory framing — so contributors know what kind of answer you're after.
The loop it closes
Spec → promote → expectedOutput → eval. You gather the ideal once, with people, then score against it automatically forever. That's the bridge between "we think it's good" and "we can prove it."