Duplicate request detection

An operations intake supervisor sees the same problem arrive more than once because people phrase the same need differently. One person files “VPN down since 9am on floor 3.” Another files “cannot connect remotely from building B.” A third forwards an email thread that never uses the word VPN. Keyword matching and exact ticket ID checks miss most of these. The queue still grows, two agents start parallel work, and the requester gets conflicting updates.

Duplicate intake is a quality problem, not only a volume problem. Redundant tickets inflate open counts, distort SLA clocks, and bury the signal that a single incident needs a coordinated response. The goal at intake is not to invent a perfect merge every time. It is to surface likely near-duplicates early enough that a human can decide before the request becomes another active work item.

Duplicate request detection treats the inbound request text as a semantic query against recent and open items in the intake corpus. An embedding model turns the new message into a vector. The system compares that vector to embeddings of open tickets, recently closed tickets in a short lookback window, and sometimes related parent incidents. High similarity scores indicate near-duplicates even when wording, spelling, or channel differ.

The useful output is a short candidate list, not a silent auto-merge. Each candidate should carry enough context for a supervisor to act: similarity score or rank, ticket ID, status, assignee if any, created time, and a short excerpt of the matched text. Ranking by similarity first, then by recency and open status, keeps the review list small. A hard cap of a few candidates prevents alert fatigue.

Channel noise matters. Email signatures, legal footers, and canned acknowledgment lines can dominate a naive embedding if they are left in. Preprocessing should strip boilerplate, normalize whitespace, and focus on the requester’s description of the problem, asset, location, and desired outcome. Structured fields already captured at intake (service, location, priority) can re-rank candidates after the semantic search, but they should not replace it. People often mis-tag category on the first pass.

Thresholds belong to operations, not to a fixed global constant. A contact-center queue with short, formulaic messages may need a higher bar than a facilities queue where long narrative emails are common. Start conservative: flag fewer, clearer matches, then widen the band once false-positive load is tolerable. Log accepted and rejected flags so threshold changes are evidence-based rather than anecdotal.

The model flags likely duplicates. Staff still merge or keep separate. That boundary keeps the system honest about ambiguity. Two tickets can be semantically close and still deserve separate handling: two printers on the same floor with different asset tags, two access requests for the same system but different employees, or a mass outage where each affected team needs its own tracking record under a parent incident.

A practical review flow looks like this. The new request arrives and is held in a short pre-queue state while similarity search runs. If candidates appear above threshold, the supervisor (or intake agent with merge rights) opens the side-by-side view. They check whether the requester, location, asset, time window, and symptoms align. If yes, they merge into the existing ticket or link as a child and notify the requester with the surviving ID. If no, they mark “not a duplicate,” which should feed back into quality review of the matcher. If unsure, they keep both open and add a cross-link so agents see each other.

Do not force a binary merge decision when the safer action is linkage. Parent-child or “related to” links preserve accountability while reducing duplicated diagnosis. The duplicate detector’s job is to make that choice visible at the front door, not to collapse every similar phrase into one record.

Empty output is the correct result when the request text is too thin to compare. A subject line of “help,” a single emoji, an attachment with no description, or a form where only mandatory dropdowns were filled does not support a reliable similarity judgment. Returning weak candidates in those cases trains people to ignore the panel.

Define minimum text length and content rules explicitly. Require a usable description field, or a parsed body after stripping signatures, before similarity search runs. If the request fails those checks, skip duplicate detection, route to normal triage or clarification prompts, and leave the duplicate panel blank rather than showing low-confidence noise. The same rule applies when the only overlapping tokens are generic service names (“email,” “badge,” “Wi-Fi”) with no distinguishing detail.

Thin text is also a cue for upstream intake design. Clarifying questions and required fields reduce both empty comparisons and true duplicates created because the first submission was incomplete and the requester filed again.

Duplicate detection sits beside other intake AI steps; it does not replace them. Auto-classification by type assigns a category and routing path. Unstructured request parsing extracts entities and fills fields. Knowledge base match at intake may resolve the need without a ticket. Duplicate detection asks a different question: is this the same open work already in flight?

Order matters. Parse enough text to compare. Optionally check knowledge articles for self-serve deflection. Run duplicate detection against the live queue before creating a new open item. Classification can still run in parallel for routing if the request is new, but merge decisions should not wait on a deep taxonomy pass. Supervisors care first about whether work already exists.

Shared failure modes appear across these steps. Boilerplate pollution hurts both classification and similarity. Over-aggressive automation that closes or merges without review creates trust debt that is hard to repair. Keep human confirmation on merges, keep empty results for thin input, and keep candidate explanations short enough to scan in a few seconds during peak volume.

Measure duplicate detection by outcomes supervisors recognize: share of new tickets that received a duplicate flag, share of flags that resulted in merge or link, share rejected as not duplicate, and time from submission to merge decision. Track reopened “false merges” separately. A rising reject rate usually means the threshold is too loose or boilerplate stripping is weak. A very low flag rate with known duplicate complaints usually means the lookback window excludes the right tickets or embeddings are built on the wrong fields.

Scope the comparison set carefully. Searching the entire historical archive creates irrelevant matches from years ago. Prefer open tickets plus a recent closed window measured in days or weeks, tuned to how long your issues typically recur. Exclude spam, phishing quarantine items, and system-generated noise tickets from the corpus so they never surface as “duplicates.”

Access control and audit trails are part of quality. Merges change requester visibility and SLA ownership. Log who confirmed a duplicate, which candidate won, and whether the action was merge, link, or dismiss. That record supports coaching and protects agents when a requester disputes consolidation.

Finally, treat the feature as a queue hygiene control owned by intake operations. Refresh embedding indexes as tickets update. Re-run similarity when a requester adds a substantial follow-up before first assignment. Keep the UI boring and fast: ranked candidates, clear excerpts, one-click merge or dismiss. The value is fewer parallel threads on the same problem, with a human still accountable for the final call.