How to build a text classifier without training a model

The work moved from data to definitions

A traditional classifier needed hundreds or thousands of labelled examples per class, and the quality ceiling was set by how consistently they were labelled.

With a capable model you define the classes in prose instead. That removes the collection bottleneck and moves the difficulty somewhere less obvious: your class definitions now carry the entire weight, and most first attempts have definitions that overlap, leave gaps, or encode a distinction nobody can apply consistently.

You still need labelled examples. You need far fewer, and they're for measuring rather than training — a hundred well-chosen items rather than five thousand mediocre ones.

Define classes so they don't overlap

Three properties, and first drafts usually miss all three.

Mutually exclusive. Every input belongs to at most one class. If "billing question" and "refund request" both fit a message about a refund charge, you will get inconsistent assignment and blame the model. Either merge them or write a precedence rule into the definitions.

Collectively exhaustive. Together they cover everything you'll receive, which in practice means including an explicit "other" or "unclear" class. Without one, out-of-scope inputs get forced into whichever class is closest, and that failure is invisible in your accuracy number.

Actionably distinct. Two classes that route to the same place and get the same treatment should be one class. Distinctions that change nothing add error without adding value.

Write each definition as what it is, what it isn't, and a borderline case with the reasoning. The borderline case is the highest-value line in the whole definition — it's where you encode the judgment that the noun alone can't carry.

Give it somewhere to put uncertainty

The single change that most improves a classifier in production: a class for "I can't tell."

Without it, ambiguous inputs are assigned confidently to a category, and you never learn they were ambiguous. With it, uncertain cases route to a human, those cases accumulate, and reading them tells you exactly what your class definitions are missing.

Expect the uncertain lane to be busy at first. That's the system working. If it's empty in week one, your classifier is guessing rather than declining, and the guesses are landing somewhere.

Ask for reasoning before the label

If the classification involves any judgment, have the model produce its reasoning first and the label second.

The order is load-bearing. Generation is sequential — a schema that emits the label first gets a category chosen with no analysis, followed by a justification written to fit it. Reversed, the reasoning informs the choice.

It also makes errors diagnosable. When a message is misclassified, the reasoning tells you whether the model misread the message or applied your definition differently than you intended. Those need entirely different fixes, and without the reasoning you're guessing which one you have.

Measure per class, not overall

Overall accuracy hides the failure that matters. A classifier that is 92% accurate can be 98% on your two common classes and 40% on the rare one that routes to legal.

Build your test set with enough items per class to say something — fifteen or so — even where that means over-representing rare classes relative to real traffic. You're measuring per-class behavior, not simulating the distribution.

Then look at the confusion pattern rather than the score. Which classes get mistaken for which tells you exactly which pair of definitions overlap, and it points at the specific text to fix.

Know when a model is the wrong tool

Not every classification problem needs one. Three cases where something simpler wins.

The rule is deterministic. If the category is decided by a field value or a keyword that reliably appears, write the rule. It's faster, free, and can't drift.

The volume is enormous and the task is simple. At millions of items, a small trained classifier on a few thousand labelled examples may be cheaper by orders of magnitude.

The distinction is genuinely subjective. If your own experts disagree half the time, no classifier will do better, and the honest fix is either to collapse the classes or to route the whole category to a person.

Watch the distribution after launch

The failure mode that arrives quietly: your class distribution shifts while your accuracy metric holds.

Real inputs change — a product launches, a policy changes, a competitor does something. The classifier keeps assigning confidently and the mix moves. Track the share of items in each class weekly, and treat a sustained shift as a signal to sample and read, not as noise.

Sample the uncertain lane on a schedule too. It's the highest-information queue you have, and its contents are the roadmap for your next definition revision.

Before you route anything on it