Skip to main content
ZeroWidth

Caliper and Langfuse

Caliper or Langfuse?

Langfuse is open source tracing, prompt management and evals that engineers can run on their own servers. Caliper lets engineers and the people who know the right answers agree on what good looks like, and scores every version against it.

In Langfuse

Dataset run: support-v8

Dataset run · 40 items · llm-as-judge: helpfulness

Each item links to its trace, with a score from the evaluator.

The evaluator template asks for a score between 0 and 1.

Nobody outside the project has looked at the answers.

In Caliper

Support answers, version 8 against version 7

  • Says what happens nextRubric from the support lead
  • Fixed 9 answers, broke 2Bigger than the noise
  • Support lead and 2 agents ratedReview by link
  • Answer key agreed by expertsFrom a spec

The support lead rated from a link with no account

Both trace and score AI. They suit different teams.

  1. 1

    Experts rate from a link

    A support lead or an outside specialist opens a review link, says who they are, and rates each answer with no account or seat.

  2. 2

    Experts can write the answer key

    A Caliper spec collects the answer each expert would give, and the one they agree on becomes the expected output your evals score against.

  3. 3

    The comparison says whether a change is real

    Comparing two runs shows which answers fixed and which broke, and a paired bootstrap says whether the overall change is bigger than the noise.

  4. 4

    Simulated customers test whole conversations

    A model plays a customer with a goal and a temperament for up to 12 turns, and the run says on which turn they got what they wanted or gave up.

  5. 5

    It sits next to the rest of the work

    Caliper runs evals each time a Workbench flow publishes, and sends the score to Ledger when a decision's deadline arrives.

When Caliper is the right choice

  • The people who know the right answer don't write code.
  • Outside experts need to rate answers without an account.
  • You need experts to agree on the right answers before you can test.
  • You build agents in Workbench and want them tested every time they publish.

Job by job

Getting traces in

Langfuse: Python and JS SDKs built on OpenTelemetry, plus framework integrations.

Caliper: One HTTP call per conversation, or an OpenTelemetry exporter pointed at Caliper.

Defining good

Langfuse: Evaluator templates, code evaluators and manual labels.

Caliper: A rubric of criteria on a numeric scale, written by whoever owns the answer.

Human review

Langfuse: Annotation queues for people in the project.

Caliper: Reviews that teammates and guests with a link rate side by side.

Prompts

Langfuse: Prompt management with versions and a playground.

Caliper: Prompts live in Workbench flows, where each publish is a version Caliper can score.

Trace in Langfuse. Judge in Caliper.

Langfuse and Caliper both take OpenTelemetry, so one agent can send the same spans to both and each team works where it's comfortable.

  • Add a second OTLP exporter, and every conversation lands in Langfuse and in a Caliper source.
  • Save the conversations worth a second look into a Caliper dataset, and send a review link to the experts.
  • Promote the answers your experts agree on, and your CI reads them back from the public API as expected outputs.
  • Connect Langfuse's MCP server to Zero, and ask about a prompt or a trace from inside ZeroWidth.

Questions

How do our traces get into Caliper?
Send them to one HTTP endpoint or point any OpenTelemetry exporter at Caliper. Your CI posts eval answers to the public API.
Can our agent send to both?
Yes. Point two OpenTelemetry exporters at Langfuse and Caliper, and each receives the same spans.

Try Caliper with your domain.

Send a review link to the person whose judgment you trust most.

No website? Describe what you do

If you'd like help with the first changes, talk to us.