Caliper and LangSmith
Caliper or LangSmith?
LangSmith traces and tests agents for engineering teams, with the deepest fit for LangChain and LangGraph. Caliper lets engineers and the people who know the right answers agree on what good looks like, and scores every version against it.
Annotation queue: refund-agent-review
Annotation queue · 25 runs · 2 reviewers
Each run shows the trace with its tool calls and feedback keys.
Reviewers work the queue from inside LangSmith.
The claims manager who knows the policy isn't in the workspace.
Refund answers, expert review
- Claims manager, outside the companyRated from a link
- Raters disagree on 5 answersHighest spread
- Judge agrees with the ratersSame rubric
- 25 conversations from the live agentFrom a source
The conversations came in over OpenTelemetry from your own agent
Both test agents. They put different people in charge of quality.
- 1
Experts rate from a link
A claims manager or an outside specialist opens a review link, says who they are, and rates each answer with no account or seat.
- 2
Experts can write the answer key
A Caliper spec collects the answer each expert would give, and the one they agree on becomes the expected output your evals score against.
- 3
The comparison says whether a change is real
Comparing two runs shows which answers fixed and which broke, and a paired bootstrap says whether the overall change is bigger than the noise.
- 4
Any framework sends the same way
Caliper reads the OpenTelemetry GenAI conventions, OpenInference and the Vercel AI SDK's spans, so LangGraph, Pydantic AI or your own loop all land as a source.
- 5
It sits next to the rest of the work
Caliper runs evals each time a Workbench flow publishes, and sends the score to Ledger when a decision's deadline arrives.
When Caliper is the right choice
- The people who judge quality don't work in your engineering tools.
- Outside experts need to rate answers without an account.
- You need experts to agree on the right answers before you can test.
- You want simulated customers to test whole conversations of up to 12 turns.
Job by job
Getting traces in
LangSmith: Python and TypeScript SDKs, native LangChain tracing, and OpenTelemetry.
Caliper: One HTTP call per conversation, or an OpenTelemetry exporter pointed at Caliper.
Defining good
LangSmith: Evaluators in code, or model judges with criteria you write.
Caliper: A rubric of criteria on a numeric scale, written by whoever owns the answer.
Human review
LangSmith: Annotation queues assigned to people in the workspace.
Caliper: Reviews that teammates and guests with a link rate side by side.
CI
LangSmith: pytest, Vitest and GitHub workflow integrations.
Caliper: Your CI posts answers to the public API, and the job fails when the score drops.
Engineers trace in LangSmith. Experts judge in Caliper.
OpenTelemetry lets one agent send the same spans to two places, so engineers keep their LangSmith traces and the people who own quality work in Caliper.
- Add a second OTLP exporter, and every conversation lands in LangSmith and in a Caliper source.
- Save the conversations an engineer flagged into a Caliper dataset, and send a review link to the experts.
- Promote the answers your experts agree on, and your CI reads them back from the public API as expected outputs.
- Run LangSmith's MCP server and connect it to Zero, and you can ask about a trace from inside ZeroWidth.
Questions
- How do our traces get into Caliper?
- Send them to one HTTP endpoint or point any OpenTelemetry exporter at Caliper. Your CI posts eval answers to the public API.
- Does Caliper work with LangGraph?
- Yes. Instrument LangGraph with OpenTelemetry and point the exporter at Caliper, and each run arrives with its model calls and tool calls as steps.
Try Caliper with your domain.
Send a review link to the person whose judgment you trust most.
No website? Describe what you do
If you'd like help with the first changes, talk to us.