How to catch prompt regressions before your users do

Why prompt changes are unusually dangerous

Code has a property prompts lack: locality. Change a function and the blast radius is roughly the callers of that function. Change a sentence in a system prompt and you have changed the behavior of every input, including the ones the sentence had nothing to do with.

The mechanism is straightforward once you have watched it happen. Adding "be concise" to fix verbose answers also truncates the ones that needed detail. Adding "always offer next steps" produces next steps on the refusals, where the correct behavior was to stop. Tightening a refusal rule makes the model decline adjacent legitimate requests.

None of these show up in the change. All of them show up in production.

Freeze a set and run it every time

Regression testing here works the way it does anywhere: a fixed set of inputs, a known standard, run on every change, compared against the last run.

The set has to be frozen. If it changes between runs, a score difference tells you nothing, because you cannot separate the effect of the prompt from the effect of the different items. Add to it deliberately, in batches, and treat each addition as a version boundary.

The set also has to contain cases that currently fail, or at least sit near the edge. A suite where everything passes has no room to detect improvement and very little to detect harm — the failures will happen in the space between your items.

For how to choose those items, the companion piece covers it in detail: start from incidents, add boundaries, add what the system should refuse, then add a quarter happy path.

Compare runs, don't read scores

An aggregate score is the wrong artifact for a regression check. Two runs can have identical averages while a third of the items moved in each direction, and the average will tell you everything is fine.

What you want is the item-level diff: which specific cases changed verdict between the last run and this one. That list is short, readable, and directly actionable — for each item that got worse, you can look at the output and decide whether it is a real regression or a scoring artifact.

The pattern to watch for is the trade. You fixed six items and broke four. The average improved. Whether that is a good change depends entirely on which four, and the aggregate cannot tell you.

Run it before the change, not just after

Half the value comes from having a baseline from the same day. Teams routinely run the suite after a change and compare against a number from three weeks ago, which has drifted for reasons nobody tracked — a model provider updated something, a retrieval corpus grew, someone edited a rubric.

Run the current version, make the change, run again. Two runs, same afternoon, same everything except the edit. That is the only comparison that isolates what you did.

This is also the cheapest guard against the most demoralizing debugging session in this field: chasing a regression for two days before discovering the baseline you compared against was itself broken.

Watch the refusals in both directions

The two failure modes here are symmetrical and teams only test one.

Over-refusal. A tightened rule makes the system decline things it should handle. Users experience this as unhelpfulness, complain less than you would expect, and quietly stop using the feature.

Under-refusal. A loosened rule — usually added to fix over-refusal — makes the system answer things it should decline. This one arrives as an incident.

Keep both in the suite permanently. The pendulum between them is the most common oscillation in a prompt's life, and without items on both sides you will fix one direction and ship the other every time.

Make the run cheap enough to actually happen

A regression suite that takes an hour and needs three manual steps will be run before big changes and skipped before small ones, and the small ones are where regressions come from.

Two things keep it in the loop. Keep the set small enough to run in minutes — this is another argument for a hundred well-chosen items over a thousand scraped. And make it runnable without ceremony, so the cost of checking is lower than the cost of wondering.

If the suite is genuinely expensive because a judge model is scoring hundreds of items, split it: a fast subset on every change, the full set before a release. A partial check that happens beats a thorough one that doesn't.

When a regression is real

You have a confirmed regression. Three options, in order of preference.

Narrow the instruction. Most regressions come from a rule stated more broadly than intended. "Be concise" becomes "keep routine confirmations to two sentences." The broad version was doing work you didn't ask for.

Split the path. If two classes of input genuinely need different behavior, stop trying to serve both with one instruction and route them.

Accept it, deliberately, in writing. Sometimes the trade is worth it. Record which items you accepted as worse and why, so the next person doesn't spend a day rediscovering a decision you made on purpose.

Before you ship the change