Embedding-based deduplication

Every multi-source search ends with the same mess. A reference manager removes the exact duplicates by identifier and title, and a pile is still left over. The same trial appears as a registry entry, a conference abstract, a journal article, and a report chapter. Nothing in those four records matches on a field.

Vector similarity works on that residue. It groups records whose text sits close together in meaning, so wording differences stop mattering. The output is a set of proposed clusters. Each carries a suggested primary record, the members, a similarity figure, and a short reason.

The saving is downstream. A pool carrying the same study five times pays five times for scoring and gives the reviewer five chances to count it twice. Independent scoring of every sub-criterion is priced per record, so duplicates are paid for in full there.

This is the most expensive piece of plumbing in triage and it pays back the least. It needs an embedding model, a store, a threshold tuned on real records, and someone to keep it working. Most rounds do not need it. It earns a place on large pools drawn from many sources, in more than one language, where hand checking has stopped being possible.

Sameness is a decision, not a property of the data. Three definitions get confused in most rounds. Same publication means two records point to one document. Same study means several documents report one piece of work. Same organization means two names refer to one body. A pipeline that blends them will fold four papers about one trial into a single row and lose three of them.

Pick one target per pass. Most evidence work wants study-level grouping, where the members stay visible and one record is marked as the report of record. Organization aliasing is a separate pass with different fields and its own review.

Name the fields that go into the vector. Title and abstract are the usual pair. Adding full text pulls in method boilerplate that makes unrelated trials look alike. Adding author lists makes one group's whole output cluster together. Every added field changes what the model treats as similar, so the field list is part of the method and belongs in the write-up.

Run this after the rules pass, not before. Removing records that fail hard criteria is cheap, and there is no reason to embed a record that is already out of scope. Where a record has no title and no abstract, the pipeline returns it untouched instead of clustering it on metadata scraps.

Similarity alone is not enough to merge on. Two records about the same intervention in the same country sit close together whether or not they report the same study. The vector step narrows the field. A second step decides.

The first step is retrieval. Embed the chosen fields, then pull each record's nearest neighbors above a loose cut. Loose is deliberate, because a pair missed here never reaches the second step. This part is cheap and can run across the whole pool.

The second step reads. A model receives both records in full and answers one question with a reason: whether they describe the same study, and which fields say so. Matching sample size, site, and enrollment window is a reason. A similar topic is not. The reason string is what the reviewer checks, and a pair with no stated reason stays unmerged.

Cluster rather than pair. Records A and B match, B and C match, and A and C do not, which happens constantly with conference abstracts. Decide the rule in advance. Either every pair in a cluster must be confirmed, or one record acts as the anchor the others are compared against. The anchor rule is cheaper, and it is why a suggested primary sits in the output.

The following round is made up, and the counts illustrate shape rather than report anything real. A hazards research center is building an evidence map on community flood early-warning systems. Four databases, two trial registries, and a sweep of humanitarian agency reports return 6,300 records. Exact-match deduplication in the reference manager removes 1,900.

The 4,400 that remain still hold repeats the manager cannot see. One program in one delta region appears as a registry record, two agency reports with different titles, a journal article, and a book chapter. The sweep of practitioner reports and registries produced most of that overlap, which is expected and not a fault of the sweep.

Title and abstract embeddings, with a loose neighbor cut, produce 810 candidate pairs. The reading pass confirms 340 of them. Those pairs resolve into 190 study families, most holding two members and a few holding five.

The pool going into scoring drops to 4,050 primaries. The other 350 records stay attached to their family and stay readable, because an agency report often carries cost figures the journal article left out. Nothing is deleted.

A second pass runs on organization names and finds 62 alias groups. One agency appears written six ways across languages and reporting years. That pass is reviewed on its own, because a subsidiary and its parent are not the same implementer.

Under-merging is annoying. Over-merging is a methods problem, because the lost record never appears in any count and nobody goes looking for it.

Related but separate work looks nearly identical to a vector. A pilot and the full trial that followed it. Two waves of one survey. Two arms of one program reported separately. Each pair shares a title style, a setting, and most of its vocabulary. Only a pass that checks dates and sample sizes tells them apart.

One global threshold does not hold across sources. Structured journal abstracts sit closer together than free-form agency summaries do, so one cut merges too much on one side and too little on the other. Tune per source, or at least check the confirmed rate per source before trusting a single number.

Merging across engines has its own trap. When the same query runs through several deep research tools, one underlying source comes back three times in three summaries. Collapsing those is right. Treating the three hits as three independent sources is the error this step exists to prevent. Counting cluster size as evidence strength brings it straight back.

Silent drift is the last one. An embedding model swapped for a newer version changes every distance in the store. Record the model, its version, and the threshold with the round.

Merges are confirmed by a person, in batches, with the reason string in view. In most evidence teams that is the information specialist who ran the searches. Clusters well above the threshold can be approved in bulk. Anything near the line is opened one at a time.

Reversibility is the design requirement. A merge sets a pointer and deletes nothing. Six weeks later a reviewer opens a primary record, sees that a pilot was folded into the trial, and splits the family apart in one action. A pipeline without that will not survive a second round.

The numbers have to reach the methods section. Reviews report how many records were found, removed as duplicates, and screened. An automated merge changes those figures, so the write-up needs the counts, the model, the threshold, and how many clusters a person confirmed.

None of this judges a record. A cluster says two documents look like one piece of work. A confirmed merge says a specialist agrees. Whether the study is any good is settled later, when a structured one-pager is compiled per candidate and somebody reads it.