Duplicate and incident-wave detection

When a payment processor goes down, the queue fills with the same problem described eleven different ways. Duplicate and incident-wave detection compares each arriving ticket against open tickets and active incidents, then attaches a similarity link where one exists. The link names the candidate parent, the score, and the fields that drove the match.

Two jobs hide under one label. A duplicate is the same customer or the same specific report already sitting in the queue. A wave is many different customers hitting one underlying fault. The response differs. Duplicates get merged. Waves get attached to an incident record so a single update reaches everyone.

Neither job includes closing the ticket. A high similarity score is a hypothesis about relatedness. Auto-close on that hypothesis and a customer whose issue merely looked similar gets a resolution notice. The problem they actually have stays open.

Speed is the point. Detection at intake, before the ticket reaches a human, is what stops fifty agents from investigating one outage in parallel. That is a different goal from clustering tickets after the fact to find shared defects, which runs on resolved history and can take its time.

Similarity needs something to be similar to. Without a live incident register, the system finds only duplicates of other tickets, which is the weaker half of the value.

The register needs four fields per incident. A plain-language symptom description, written the way customers describe it rather than the way engineering does. The affected scope: regions, product areas, versions, or customer segments. A start time. And a status: investigating, identified, monitoring, or resolved.

Write symptom text for matching, not for the status page. "Card payments declining at checkout" matches customer language. "Elevated errors on the authorization service" does not, and no arriving ticket will resemble it. Keep both, and match on the first.

Scope is the guard against over-matching. An incident limited to one region should not absorb tickets from another. If scope is unset, treat the incident as unscoped and discount every match it produces.

Define the thresholds and their consequences up front. A high band can auto-link and suppress a duplicate alert. A middle band attaches a suggestion an agent confirms. Below that, do nothing. If no incident is open and nothing clears the high band, the ticket proceeds as normal. Silence is the correct output most of the time.

Embedding similarity on ticket text is the base signal, and alone it is not enough. Two tickets can describe the same symptom in different products. Two tickets from one customer minutes apart are probably a single issue even when the wording diverges.

Carry at least four signals and keep them in separate columns. Text similarity between the new ticket and the candidate. Account and identity overlap, which is what distinguishes a true duplicate from a wave member. Time proximity, because a match against a ticket from March is a different claim than a match against one from twenty minutes ago. And attribute overlap: same product area, region, version, or error code.

Blending these into a single number hides which one fired. An agent reviewing a suggested merge needs to see whether it rests on account overlap plus a five-minute gap, or on text similarity alone. The second is much weaker and should be labeled that way.

Arrival rate deserves its own treatment. When several tickets matching one symptom land inside a short window, that pattern is itself the signal. It can fire before any incident record exists. Surface it as a possible emerging wave for a human to confirm. Do not create the incident automatically.

Take a hypothetical Tuesday. A point-of-sale vendor serves independent restaurants across several states. At 6:40pm, card payments start failing at terminals in one state after a regional acquirer link degrades.

The first ticket arrives at 6:44pm by phone. A diner manager says cards are being declined and the queue is out the door. No incident exists, no strong match is found, and the ticket routes normally.

Three more arrive by 6:51pm. All describe declines, all come from the same state, all mention card present transactions. Text similarity is high, account overlap is zero, and the arrival window is eleven minutes. The system raises an emerging wave suggestion with the four tickets attached.

A support lead checks the acquirer status and opens an incident. Symptom: card payments declining on terminals. Scope: one state. Status: investigating. Matching now has a target.

By 7:20pm, thirty-one more tickets have arrived. Twenty-six clear the high threshold and receive the incident's holding update. Five land in the middle band and go to an agent. Three are the same outage and get linked by hand. Two are not. One is a printer fault, and one is a terminal that was never provisioned.

Those two are why the middle band exists. An auto-close rule would have told a restaurant with a broken printer that its payment problem was being worked. When the acquirer link recovers, one update goes out, and a follow-up asking whether the issue is resolved handles the linked tickets as a set.

The expensive error is not a missed link. It is a confident wrong one.

Merging two problems hides the smaller one. During a large incident, unrelated tickets get swept in because they use the same words. Those customers wait for a fix that will never apply, then close alongside the incident. Keep merged tickets individually addressable, and confirm resolution per ticket rather than per incident.

Over-broad incidents pull in everything. An incident described as "app not working" with no scope will match most of the queue. Narrow symptom text and real scope are the only defenses.

Stale incidents keep matching. One left in monitoring for a week absorbs new tickets that describe a different fault in similar words. Expire incidents on a timer and make a human extend them.

Silent linking costs trust. A customer whose ticket was attached and never acknowledged does not know anyone saw it. Every linked ticket still gets a reply, even if that reply is the standing update.

Volume counts become claims. Tickets linked to an incident count contacts, not affected customers. Most people who hit an outage never write in. When that number travels to engineering as a filed bug cluster, label it as contacts.

Two humans hold decisions here, and they are not the same person.

The incident commander decides that an incident exists, what its scope is, and when it ends. A clustering signal can raise the suggestion. It should never open, widen, or close an incident by itself, because the scope decision determines who gets told what.

The agent decides each merge below the high band. They see the candidate parent, the signals that matched, and both ticket texts. They confirm, reject, or split. Record the rejections. Repeated rejections against one incident usually mean its symptom text is too greedy.

Priority stays outside this system. A ticket linked to an incident is not automatically lower priority. A restaurant losing an evening of card payments has a real deadline whether or not fifty others share the fault. That judgment sits with forecasts of which tickets will breach SLA and the people who read them.

Review after each large wave. Pull the tickets that were linked and later split back out, then read what they had in common. Those shared strings are what to remove from the next incident description.