Auto-knockout filter

Evidence teams arrive at triage holding a pool nobody can read in the time available, against criteria that were agreed weeks earlier. A knockout filter handles the part of that pool where no judgment is needed. It applies the stated hard criteria as code, drops the records that fail, and writes the failing rule on every drop.

The line to hold is between a rule and a rating. A rule returns true or false from a field: published before the cutoff year, a country outside scope, a record type the protocol excludes. A rating is a judgment about relevance or quality, and it belongs to the step that runs next. Anything that requires weighing two considerations against each other is not a knockout.

Cost is why this runs first. Model scoring is priced per candidate, and a pool carrying thousands of records that are ineligible on their face pays that price for answers nobody reads. Scoring each sub-criterion independently earns its cost on survivors. It is waste on a record that failed a date test.

The filter holds no opinion about hard cases. If a rule cannot be evaluated because the field is empty, the record does not fail. It moves to a pile a person works through by hand.

A rule needs three parts before it runs. Name the field it reads. State the test that field must pass. Say what happens when the field is empty. A criterion sitting in the protocol as prose is not yet a rule, and "recent studies only" names no cutoff, no field, and no behavior on a record with no date.

Hard criteria come from the scope stage. If the protocol never named a design restriction, triage does not add one because the pool turned out large. Take the criteria as the translation of the brief into a hypothesis tree and screening criteria left them, and keep the wording. A criterion that cannot be implemented against any available field is a finding for the protocol owner, not permission to approximate.

Test the rule set before pointing it at the pool. Run each rule against a small set of records the team already classified by hand, including several they consider central. A rule that removes a paper the reviewers expect to see is either coded wrong or exposes a criterion that was wrong. Stress-testing the criteria themselves sits upstream of this; here the check is narrower, on field names and thresholds.

Version the rule set and attach the version to the round. The question six months from now will be which rules were live when a particular record was cut, and a rule set edited in place cannot answer it.

The filter produces three outcomes, not two: pass, fail against a named rule, and unresolved. Unresolved means the record lacked the field the rule needed. Treating unresolved as failure is the quiet way a filter turns into a bias.

Metadata completeness varies by source in a patterned way. Journal records from a major index usually carry a year, a document type, and a structured abstract. Registry entries, ministry reports, conference abstracts, and working papers often carry one of those and sometimes none. A filter that rejects on absence removes the least indexed material first, which is exactly where the grey literature and practitioner sweep spent its effort.

Set the default per rule rather than globally. A record with no publication year cannot be tested against a date cutoff, so it routes to review. A record with no country field in a review restricted to two countries also routes to review, because the study may still be in scope. A document type field that is present and reads "erratum" is a clean fail.

Cap the unresolved pile before the run. If more records land in manual review than the team can process, the filter saved nothing. That usually means a rule depends on a field the corpus does not carry reliably, and it needs a different field or a different test.

The review below is imagined, and its record counts are there to show the shape of the work. An evidence synthesis team is reviewing school meal programs and their effect on school attendance in low and middle income countries. Searches across four databases, one trial registry, and a targeted agency sweep return 4,100 records after the reference manager removes exact duplicates.

The protocol carries four hard criteria. The study reports an attendance or enrollment outcome. The setting is a low or middle income country. The design is an evaluation with a comparison group. The record is a study rather than a commentary, editorial, or unfinished protocol.

Only two of those are codeable at this point. Document type is a structured field in three of the four databases. Publication window is a date field. Country and design live in free abstract text, and abstracts describe design loosely enough that a rule keyed on the word "control" would cut real trials and keep several modeling papers.

So the filter runs two rules. Document type removes 380 editorials, errata, and registered protocols with no posted results. The date cutoff, set in the protocol at 2005, removes 610 more. Another 240 records carry no usable document type and route to the unresolved pile, most of them from the agency sweep.

That leaves 2,870 records for scoring and 240 for a person to open. An afternoon on the rule set cut roughly a quarter of the pool before any model read an abstract. Country and design stay where they belong, in the scored assessment.

These failures are cheap to create and expensive to notice, because a knockout leaves no score behind to inspect.

Keyword rules pretending to be field rules do most of the damage. "Reject if the abstract does not contain randomized" looks like a design filter. It removes every trial that spelled the word differently, every trial that described allocation without using it, and every record with no abstract at all. Rules read fields. If the criterion needs meaning, it is not a rule.

Silent scope drift is the second one. Someone adds a rule mid-round because the pool is still too big, and the drop log now mixes protocol criteria with convenience. Any rule that was not in the protocol has to be labeled that way, and it should be a decision a named person made rather than a threshold someone nudged.

Dirty source fields break rules that are correctly written. A date field holding the online-first year in one database and the issue year in another will cut differently on either side of the same cutoff. Check each rule against each source separately before trusting the combined count.

The last failure is treating the drop pile as gone. It is a population with a known composition, and it can be sampled. Re-scoring dropped candidates with an advocate persona is the cheap way to learn that a rule was cutting the wrong thing.

The person accountable for the review owns the rules, in writing, before the run. Approval means reading each rule as coded, not approving the protocol a second time. The gap between "studies since 2005" in a protocol and a test on one specific date column is where the disagreements live, and they settle faster on a page than in a drop log of 990 records.

The drop log has to be readable. Every removed record needs its identifier, its title, and the rule that removed it. A log that says only "excluded at triage" cannot be audited and cannot be written into a methods section. Counts by rule belong on the same page, because a rule that removed four records and a rule that removed six hundred deserve different attention.

The unresolved pile is human work, not a backlog to clear with a second automated pass. Someone reads those records and makes a call. Those calls are the raw material for fixing the rule set next round.

Nothing in this step is a finding. A knockout filter reports that a record did not meet a stated condition. It does not say the study was weak, and it does not say the evidence base is thin. Those claims come later, from people who read what survived.