LLM-scored QA on every ticket

Manual QA reads a handful of tickets per agent per month. The sample is small enough that an agent can go a whole quarter without a single reviewed conversation. Full-coverage scoring reads every closed ticket and returns a mark per criterion, with the line of transcript that justified it.

The output is three things. A scorecard per ticket, a distribution across the team, and a shortlist of tickets a human should read. The third one is the point. Finding the twelve tickets worth reviewing beats producing four thousand numbers.

A score is a reading of a transcript. It is not a measure of a person. It cannot see the phone call that preceded the email. It cannot see the outage that made the tooling unusable, or the customer who arrived already angry.

That gap is why scores stay inside a review process. Numbers about named people travel fast and shed their caveats on the way. Keep the rubric and the evidence attached wherever the number goes.

The rubric is the whole system. A model scoring against vague criteria produces a confident number with nothing underneath it. Write each criterion as a statement a person could check against a transcript and agree on.

Testable looks like this. The agent stated the next step and a timeframe. The agent acknowledged the customer's stated problem before offering a fix. Professionalism and ownership are not criteria, because two reviewers will score them differently.

Agree the weights with agents in the room. A rubric written only by managers gets treated as a trap. When agents help set what counts, the first month of scores starts an argument about the work rather than about the instrument.

Set the abstention rule before scoring starts. When a transcript holds no evidence for a criterion, the model returns not evidenced rather than a low mark. Scoring absence as failure is the fastest way to lose the team.

Split rubrics by channel. A chat transcript and a phone summary carry different evidence, and an email rubric will punish the medium rather than the agent. Exclude tickets that ran mostly by phone unless a written summary exists.

Score each criterion separately and quote the text that supports it. A ticket-level total works as a rollup, as long as it can be opened and read as parts. An unopenable number cannot be coached against or contested.

Keep the scale coarse. Met, partly met, not met, not evidenced. Fine point scales imply a precision the model does not have.

Attach a confidence value and route low confidence to a human. Long threads, mixed languages, and tickets with three agents on them are where the model is least reliable. Those belong in a review queue, not in an aggregate.

Calibrate monthly against human scores. Take a set of tickets, have two QA reviewers mark them by hand, and compare. When the model and the reviewers disagree in a consistent direction, fix the rubric wording before blaming either side.

Some criteria are better handled before a reply goes out. Whether a message promised something the company cannot deliver belongs to an adversarial review of the drafted reply, which prevents the problem instead of grading it.

Treat the following account as illustrative only. A regional broadband provider runs a support team of nine agents covering billing and connectivity. In one month the team closes about 4,100 tickets, of which roughly 1,600 are billing.

Manual QA had reviewed four tickets per agent per month. Full scoring covers all 4,100 against a six-criterion rubric. The distribution is unremarkable except for two findings.

One criterion scores low across the entire team: stating when a credit will appear on the bill. That is not nine coaching conversations. It is a missing sentence in the credit macro, and the same gap shows up in replies drafted from knowledge-base sources.

The second finding is about one person. An agent named Priya sits well below the team on that criterion in connectivity tickets. Her manager opens six of her flagged tickets before saying anything to her.

Reading them changes the picture. Priya handles the escalations nobody else takes, where the next step depends on a field visit dispatch has not scheduled. She cannot state a timeframe that does not exist. She disputes three of the six marks.

The other three are fair, and the coaching is specific. Say that dispatch will confirm within two business days, even when the visit date is unknown. That is one conversation about six tickets, not a rating.

Length bias is the most common. Longer replies contain more evidence, so they score better. An agent who solves a problem in two sentences can look worse than one who writes five paragraphs and solves nothing.

Template use inflates scores. Macros written to satisfy the rubric will satisfy the rubric. Watch for criteria that pass on nearly every macro-heavy ticket, then check whether the sentence actually applied to that customer.

Hard tickets get punished twice. The agent who takes angry, complex, multi-day cases will show lower satisfaction and lower rubric marks. Report scores alongside case mix, or the team learns to avoid the hard queue.

Phrasing differences read as quality differences. Agents writing in a second language, or writing plainly rather than warmly, lose points on criteria that are really about style. Keep style out of the rubric unless the company can define it in testable words.

Rubric changes break comparison. A mark from March under one rubric and a mark from June under another are not the same measurement. Version every rubric, and never chart across a version change without labeling it.

Some signals belong outside conduct scoring entirely. A ticket that came back is worth understanding. Prediction of which closed tickets are likely to reopen does that job without grading the person who closed it.

A model score may be used for a short list of purposes. Choosing which tickets a human reviews. Finding criteria that fail team-wide. Spotting a broken macro or missing help content. Giving an agent a private view of their own work.

It must never be the sole basis for anything touching employment. No termination, no formal warning, no pay decision, no ranking, no queue or shift assignment from a model score alone. Each of those requires a human who has read the tickets and can say what happened in them.

Write that rule down and name who enforces it, usually the support director. When a score enters a performance conversation, the manager reads the cited tickets first and brings them to the meeting.

Agents see their own scores and the evidence before any manager acts on them. Hidden scoring turns this into surveillance. The criterion, the mark, and the quoted line belong on the same screen.

The dispute path has to be cheap. One click on a criterion opens a contest with a box for the agent's account of what happened. A reviewer who is not their manager re-reads the ticket and rules within a few days. The decision, the reason, and the reviewer's name go on the record.

Disputed marks leave the aggregate until the dispute is settled. Overturned marks stay visible in the record, because a pattern of overturns says something about the rubric rather than the agent.

Dispute volume is itself a rubric measure. When one criterion generates most of the contests, the criterion is badly worded, and the QA lead rewrites it. Confirmed strong tickets are worth routing too, since they feed the expertise graph that records who resolves what.