Intent and category classification

Support ops leads want one thing from a classifier at triage: a category that sends the ticket to the right queue. The output is a label, a confidence value, and the text span that drove it. It is not an explanation of what broke. A ticket labeled billing has not been diagnosed as a billing problem. It has been pointed at the people who handle billing.

That distinction matters downstream. Routing rules, macros, and reporting all read the same field. If the label carries more meaning than the taxonomy defines, every consumer of that field inherits a claim nobody made. Keep the category narrow enough that a new agent can apply it the same way twice.

Zendesk Intelligent Triage and SentiSum sit in this class of tools. They predict fields on an existing ticket. Neither one designs your category list, and neither knows which queue is short-staffed this week. The taxonomy is a support ops artifact. The model only applies it.

Classification also assumes the ticket already holds enough text to read. When intake arrives as a two-word chat message, the classifier has nothing to work with. Filling those gaps belongs upstream, with follow-up questions that populate required fields before submission.

A classifier can only return labels somebody wrote down. Define the list first, with a one-line rule for each category and two real tickets that belong in it. Categories without examples drift within a month, because two annotators will read the name differently.

Four decisions have to be made before training, and support ops owns all four. How many categories the list holds, and whether they are mutually exclusive. Whether a ticket may carry a second label. What happens when the top prediction is weak. Which categories are allowed to trigger automation on their own.

The fallback category is the one people forget. Every taxonomy needs a destination for tickets that fit nothing, and that destination must be a real queue with a real owner. An unrouted bucket nobody reads is worse than no classifier, because the ticket looks handled in the dashboard.

Training data comes from tickets your team already labeled. Sample from the last few months, not from the whole archive, and drop labels applied by bulk edits. If a category has only a handful of clean examples, leave it out of the model and route it by keyword until the volume exists.

When the taxonomy is unwritten, do not run the model. A label list invented by an LLM on the fly will look reasonable and will not match the queues, the macros, or last quarter's report.

Every prediction arrives with a score. Treat that score as a gate, not as a display value. Above the threshold, the label writes to the ticket and routing fires. Below it, the label is a suggestion an agent confirms, and the ticket lands in the general queue meanwhile.

Set thresholds per category, not globally. A password reset label can auto-route at a lower bar because the cost of a mistake is one transfer. A label that sends a ticket to the security or legal path should require near certainty, or should never auto-route at all.

Multi-label output needs its own rule. A customer who reports a failed charge and asks to cancel is two jobs in one ticket. Decide in advance whether the classifier splits the ticket, picks the primary intent, or flags it for a human to split. Silent primary-label selection hides the second request until the customer follows up angry.

Keep the label separate from the queue. Categories change slowly. Queue structure changes when a team reorganizes. If routing reads the category through a mapping table, a reorg is a table edit rather than a retrain. Downstream steps such as matching ticket content to agent skill profiles then consume a stable field.

This walkthrough is hypothetical. A payroll software vendor runs a taxonomy of fourteen categories. Two of them sit close together: reporting and export, which covers files the customer generates, and tax filing, which covers submissions the vendor sends to an agency.

Ticket A reads: the quarterly wage report downloads as an empty CSV for our two newest locations. The classifier returns reporting and export with high confidence, cites the phrase empty CSV, and routes to the reports queue. The threshold for that category is set low because a wrong route costs one reassignment.

Ticket B reads: our Q3 filing came back rejected and the report we pulled shows the same missing locations. The words report and pulled push the model toward reporting and export. Confidence lands under the threshold, because filing language appears in the same sentence. The ticket goes to the general queue with both candidate labels shown.

A triage agent reads Ticket B in under a minute and picks tax filing. That category has a regulatory deadline attached, so it never auto-routes. The agent's correction is stored with the original prediction, and it becomes a training example on the next cycle.

Ticket C arrives twenty minutes later from a different customer, with almost the same wording as Ticket B. That similarity is a separate signal, handled by matching new tickets against active incidents at intake. The classifier still labels each one independently. Category and incident membership are different fields, and merging them costs you both.

Accuracy at launch tells you very little about month six. Four things degrade a working classifier, and none of them announce themselves.

New products create tickets with no category. A launch introduces vocabulary the model has never seen, and those tickets get forced into the nearest old label. Watch the fallback rate around every release. A sudden drop in unclassified volume is suspicious, not reassuring.

Agent corrections stop being recorded. Once agents learn that fixing a label changes nothing, they reroute the ticket and leave the field wrong. The training set then reflects the model's own past output. Make the correction one click, and show agents where their corrections went.

Rare categories look accurate and are not. A label that fires on a small number of tickets each quarter can appear healthy in an overall accuracy number while missing most of its real cases. Report per-category performance, and be willing to retire a category that never earns its place.

Channel differences get ignored. A phone transcript, a forum post, and a formal email describe the same problem in different registers. If the model trained mostly on email, chat accuracy will lag quietly. Converting every channel into one ticket schema reduces the gap but does not erase it.

Any agent can change a category on any ticket, with no approval and no justification field. That override is the safety valve for every threshold decision above it. Making agents ask permission to correct a label guarantees the label stays wrong.

Adding or removing a category is a different decision, and it belongs to the support ops lead who owns reporting. One person approves taxonomy changes, versions them with a date, and tells the analytics team. Otherwise two managers add near-identical labels and the quarterly trend breaks.

Review the correction log every week rather than the accuracy dashboard. Corrections point at specific confusions between specific pairs of categories. Those pairs are usually a definition problem, not a model problem, and the fix is a clearer rule rather than more training data.

The category never sets the ticket's place in line on its own. It says which team should look. How urgent the ticket is comes from separate signals, and combining the two into a single automatic decision removes the person who should be weighing them. Priority work is handled by a score built from sentiment, tier, and language cues, and a manager can move any ticket regardless of either field.