Skip to main content
ZeroWidthdocs
Get started

Part 3 — Prove it holds up

Our classifier from Part 1 runs from code as of Part 2. But "it looked right in the test pane" isn't proof. Now we measure it in Caliper, so we can tell a real regression from a lucky demo — and catch one before it ships.

Build the answer key

  1. Make a dataset of tickets with their correct queue

    In Caliper, create a dataset. Each item is a ticket (input) plus the queue it should land in (expectedOutput) — the golden answer the judge scores against:

    inputexpectedOutput
    Card declined twice but I was still charged.billing
    Export button throws a 500.technical
    Whole team locked out, launch in an hour.urgent

    Don't cherry-pick the easy ones — the value of the set is the tickets your classifier is likely to get wrong. Twenty honest cases beat two hundred obvious ones. (Shortcut: Caliper can generate a starter set from your flow, which you then correct.)

  2. Write a rubric

    A rubric is what "good" means. For a classifier it's simple — one criterion, "Correct queue": did the output match the expected queue? Start from the Quick reactions preset (a pass/fail) and rename it, or write the one criterion yourself.

Score it

  1. Create an eval that runs your flow

    Create an eval: pick the dataset + rubric, and set the target to your Workbench flow. That means Caliper runs the classifier itself on each ticket, then judges the result against the expected queue — no code needed on your side.

  2. Run it and read the score

    Kick off a run. You get an overall score plus a per-item, per-criterion heatmap. The score tells you how good; the heatmap tells you where it's weak — the red rows are the tickets it's misrouting.

  3. Find and fix a failure

    Say account and urgent are the reds — the classifier sends "locked out, launch in an hour" to account instead of urgent. That's a category-boundary problem. Go back to Part 1, sharpen the urgent description ("time-critical regardless of topic"), republish, and re-run the eval. The cell going green next run is your proof the fix worked.

Make it a gate, not a one-time check

A score you look at once is a demo; a score that runs on every change is a guardrail. When you're changing the flow from CI, wire the external eval loop: your CI produces the classifications, submits them, and fails the build if the score drops below your bar.

For the curious — why an LLM judges an exact-match classifier

For a single-label classifier the judge is mostly confirming actual == expected, which feels heavy. The reason to keep it in Caliper anyway: the same dataset + rubric machinery scales to the fuzzy cases (tone, helpfulness, faithfulness) where there's no exact match and you genuinely need judgment — so your triage classifier and your support-reply agent live under one measurement system. The judge attaches its reasoning to every score, so a wrong call always comes with a why.

Polling for the CI score

There's no public run-poll endpoint yet. After submitting an external run, re-fetch the eval with GET /1.0/caliper/evals/:id and read latestScore. See the CI guide.

What you built

A support-ticket classifier that's built (Part 1), callable from your code (Part 2), and measured on real tickets with a gate against regressions (Part 3). That's a production AI feature.

And the gap the eval exposed is the next thing to build. That's the improvement loop: what you measure points at what to fix, and the loop turns. From here, Compass is where you find the next place AI belongs.

4 min read