Caliper and Braintrust
Caliper or Braintrust?
Braintrust gives engineers SDKs, scorers and experiments for testing AI in code. Caliper lets engineers and the people who know the right answers agree on what good looks like, and scores every version against it.
support-bot · experiment 2026-10-14
Experiment · 40 rows · Factuality, custom scorer
Factuality rose 4 points against the base experiment.
The scorer is a prompt that an engineer wrote in the repo.
The support lead hasn't seen the results.
Support answers, version 8 against version 7
- Gets the policy rightRubric from the support lead
- Fixed 9 answers, broke 2Bigger than the noise
- 3 raters agree with the judgeReview by link
- Answers from your CIExternal eval
The support lead wrote the rubric and rated from a link
Both score AI. They serve different people on the team.
- 1
Experts rate from a link
A support lead, a lawyer or an outside clinician opens a review link, says who they are, and rates each answer with no account or seat.
- 2
Experts can write the answer key
When nobody has the right answers yet, a Caliper spec collects what each expert would say, and the answer they agree on becomes the expected output.
- 3
The comparison says whether a change is real
Comparing two runs shows which answers fixed and which broke, and a paired bootstrap says whether the overall change is bigger than the noise.
- 4
Simulated customers test whole conversations
A model plays a customer with a goal and a temperament for up to 12 turns, and the run says on which turn they got what they wanted or gave up.
- 5
It sits next to the rest of the work
Caliper runs evals each time a Workbench flow publishes, and sends the score to Ledger when a decision's deadline arrives.
When Caliper is the right choice
- The people who know the right answer don't write code.
- Outside experts need to rate answers without an account.
- You don't have an answer key yet and need experts to agree on one.
- You build agents in Workbench and want them tested every time they publish.
Job by job
Getting traces in
Braintrust: SDKs in six languages, OpenTelemetry, and a CLI.
Caliper: One HTTP call per conversation, or an OpenTelemetry exporter pointed at Caliper.
Defining good
Braintrust: Scorers written in code or as prompts, including the autoevals library.
Caliper: A rubric of criteria on a numeric scale, written by whoever owns the answer.
Scoring
Braintrust: Code scorers, model judges and human review.
Caliper: A model judge and people score against the same rubric.
Outside reviewers
Braintrust: Braintrust doesn't advertise rating by link with no account.
Caliper: Anyone with a review link rates without an account.
Engineers in Braintrust. Experts in Caliper.
Your agent's OpenTelemetry exporter can send the same spans to both, so engineers keep their experiments and the people who judge quality work in Caliper.
- Add a second OTLP exporter, and every conversation lands in Braintrust and in a Caliper source.
- Save the conversations an engineer flagged into a Caliper dataset, and send a review link to the experts.
- Promote the answers your experts agree on, and your CI reads them back from the public API as expected outputs.
- Connect Braintrust's MCP server to Zero, and ask about an experiment from inside ZeroWidth.
Questions
- How do our traces get into Caliper?
- Send them to one HTTP endpoint or point any OpenTelemetry exporter at Caliper. Your CI posts eval answers to the public API.
- Which agents can Caliper test?
- Any agent that sends its conversations by HTTP or OpenTelemetry shows up as a source in Caliper, wherever it was built.
Try Caliper with your domain.
Send a review link to the person whose judgment you trust most.
No website? Describe what you do
If you'd like help with the first changes, talk to us.