Customer-reply intent classifier

Follow-up replies arrive faster than agents can read them, and most of them say very little. A classifier reads each reply and picks one of three labels: resolved, not resolved, or unclear. The label decides what happens to the ticket next. It says nothing about what the fix was.

Resolved means the customer stated that the problem is gone. Not resolved means the customer says it is still happening, or that a new problem appeared in the same thread. Unclear covers everything else, and everything else is a large pile. A one-word thanks belongs there. So does a reply that answers a different question.

The third label carries the weight. A two-label model has to push every ambiguous reply onto one side, and it will pick resolved. On the surface, resolved is what most replies look like. Unclear is not a failure of the model. It is the honest output for a message that settles nothing.

The classifier never closes a ticket. It sets a state and routes. Resolved sends the ticket to a pending-close queue. Not resolved reopens it and returns it to the assigned agent. Unclear goes to a person for ten seconds of reading.

The definitions belong to the support org, not to the vendor. Write them as sentences an agent can apply by hand. If two experienced agents label the same fifty replies differently, no model will do better than that disagreement.

Decide the hard cases in advance. A customer who thanks the agent and then asks about billing is not resolved. A customer who says the workaround is acceptable but the bug remains is also not resolved. A customer who ignores the issue and asks for an invoice copy is unclear on this ticket, and probably a new ticket besides.

Set a confidence floor and a default. Below the floor, the reply gets unclear, whatever the top score was. That single setting is what keeps a quiet model from closing real work. Teams tend to pick a floor once and never revisit it. It should move whenever the definitions move.

Language coverage needs a decision too. A team handling several languages either runs a multilingual model or translates first, and both options lose something. Where coverage is thin, send those replies to unclear by default rather than scoring them badly.

Some replies should skip the model entirely. Out-of-office notices, bounce messages, and delivery receipts are not customer replies. Filter them by header before classification runs. A vacation auto-reply labeled resolved is a closed ticket that nobody read.

Openings mislead. People start with thanks because that is how people write. The signal usually sits in the second half, after a but or a however. Feed the classifier the full message text, every time.

Give the model the question it is answering. The same reply means opposite things depending on what was asked. Yes answers "is it working now" and "do you still see the error" in different directions. Pair every reply with the last agent message before scoring it.

Channel changes the input. Chat replies are short and clipped. Email replies carry signatures, quoted history, and forwarded threads. Strip the quoted history first, or the model will score last week's complaint a second time.

The model reads text, and only text. It should not weigh how fast the customer replied or how many tickets they filed last quarter. Those signals belong to other models with other jobs. Mixing them in produces a label nobody can explain to a customer who disagrees with it.

Return three things per reply: the label, a confidence, and the span of text the label came from. The span is what makes review fast. An agent checking a disputed label wants the sentence, not a number.

The following case is constructed for illustration. A company sells home security cameras with a subscription app for clip storage. Support receives several hundred follow-up replies a week across email and chat, and each one waits for an agent to open it.

Take four replies from one morning. The first says the camera reconnected after the firmware update and the night view looks normal again. That is resolved at high confidence, and the span is the second clause.

The second says thanks for the help, and nothing else. The classifier scores resolved, but weakly, so the score lands under the floor and the label returns as unclear. That is correct. The customer never mentioned the camera.

The third says the doorbell chime works now, but clips still stop recording after four seconds. Two claims sit in one sentence, one positive and one negative. The written rule says an open complaint outweighs a reported fix, so the label is not resolved, and the span quotes the clip clause.

The fourth is a forwarded email with three quoted messages and one new line asking about the refund policy. After the quotes are stripped, the new line says nothing about the camera. Unclear on this ticket, with a flag for a second request.

One reply advances on its own, one reopens, and two reach a person. That split is the realistic shape of a morning. A model that routes almost nothing to humans is guessing on the rest.

One failure dominates the others. Politeness reads as resolution. Customers close warmly while the product is still broken, and models trained on general text absorb that habit. Review errors in this direction first, and treat them as more costly than the opposite mistake.

Sarcasm scrambles the label. "Great, so it broke again" contains a compliment word and a failure report. Sentiment and resolution are separate axes. Keep them apart, and let the tone read feed priority instead while the classifier reads the claim.

Partial fixes get flattened. A ticket with three reported symptoms and one fixed symptom is not resolved. Where tickets routinely carry several complaints, label per issue rather than per reply, or the model will keep rounding up.

Silence is not a label here. A customer who never writes back produces no input at all, and this model has nothing to score. That case belongs to the separate decision about how long silence may run, which reads ticket history rather than message text.

Drift arrives with wording changes. When the outgoing follow-up question gets rewritten, replies change shape, and old label behavior stops matching. Sample a batch by hand after any change to the message.

No label closes a ticket on its own. Resolved moves the ticket into pending close with a waiting period attached. During that window the customer can write again, and a later reply always outranks the earlier label.

Not resolved acts faster, and it should. It reopens the ticket at once, keeps the original agent, and preserves the ticket number and the whole thread. The customer never re-explains anything. That rule is the main reason the label is worth having.

Unclear goes to a queue with a target time, not to a folder. Someone reads it the same day. An unclear pile with items older than a few days means the floor sits too high. It can also mean the definitions are too narrow for real messages.

Review the misses monthly, using reopened tickets as the source. Every ticket that was labeled resolved and came back names a reply the model read wrong. Feed confirmed outcomes into the closing summary that records what actually fixed it. Send the misses to the model that predicts which closed tickets return.