Adversarial cull-check

Triage throws away most of what it touches, and the thrown-away pile is the one part of a research round nobody reads again. This check reads a sample of it. A model is given a dropped candidate and asked to make the strongest honest case that it belonged in the shortlist. A second step tests that case against the criteria the round already published.

The output is a short list of contested drops with the argument attached. It is not a re-score, and it does not put anything back. Each contested row goes to the person who owns the round, who either reinstates the candidate with a reason or records why the original cut stands.

Most of the value is information about the filter rather than about the candidates. A round where the advocate finds nothing tells you the cut was clean. A round where the advocate keeps winning on the same criterion tells you that criterion is misfiring, and that finding is worth more than the two or three candidates recovered.

Keep expectations proportionate. This is a sampling audit on work that has already been done, run on a fraction of the drop pile, and a good round produces a handful of contested rows. Treating it as a second full pass over everything cut converts a cheap safeguard into a second triage.

Draw the sample from strata, not from the pile as a whole. Group the drops by why they were dropped: failed a hard rule, scored low on one criterion, scored low across the board, or fell just under the shortlist line. Sample within each group. A flat random draw over thousands of records will mostly return obvious rejects, and the advocate will lose every time in a way that teaches nothing.

The near-miss stratum is the least interesting one, which surprises people. Candidates a point under the line were already seen by the process working as designed. The productive strata are the rule-based cuts, where a coding error removes a whole class silently, and the single-criterion cuts, where one weak cell sank a candidate that was strong everywhere else.

The advocate needs the same material the scorer had. Hand it the candidate record, the published criteria, and nothing about the score it received. Showing the model the original score invites it to argue with the number instead of arguing from the evidence, and a model that knows a candidate was rejected tends to produce a defense shaped around that rejection.

Fix the sample size and the strata before the run. A sample resized after the first few results is no longer a sample, and the recovery count it produces cannot be compared with the previous round.

The persona has one job: build the best case this candidate meets the stated criteria, using only what is in the candidate record. It may reinterpret ambiguous evidence generously. It may not add facts. A defense that rests on a study probably having a comparison group, or an organization likely having capacity, fails at the next step and should be marked as unsupported rather than counted as a contest.

Run the reply through the criteria in a second pass, with the advocacy framing removed. That pass asks a narrow question per criterion: does the cited evidence meet the written standard. A case that survives both steps is a contested drop. A case that reads well and cites nothing checkable is an argument, and arguments are not findings.

Separate the two possible verdicts on a contested row. Either the candidate met the criteria and the scoring missed it, or the candidate did not meet the criteria and the criteria are wrong. Those need different fixes. The first is a scoring problem for independent scoring of each sub-criterion to absorb. The second belongs upstream with criteria stress-testing, because a criterion that keeps excluding useful work was badly specified.

Log the cases the advocate lost as well. A rejected record that survived a determined defense is stronger evidence that triage worked than any pass rate, and the losing arguments show which parts of the criteria are doing the actual work.

The example is constructed, and every number in it is invented. A policy analysis unit is reviewing evaluations of homelessness prevention programs to advise a funding decision. Sourcing returned 1,240 records. Triage kept 96 and dropped the rest.

The drop pile splits into three strata. Around 700 failed a rule on document type or date. Around 380 scored low across every criterion. Around 60 were dropped on a single criterion, most often the requirement that the evaluation report a housing stability outcome measured at least six months out.

The team samples 40: fifteen from the rule stratum, fifteen from the low-scoring stratum, and ten from the single-criterion stratum. The advocate wins nothing in the low-scoring stratum, which is the expected result. It produces four contested rows in the rule stratum and three in the single-criterion stratum.

Six of the seven are readable in one sitting. Four are municipal evaluation reports that carry no document type in the source record and were removed by a rule that treated a missing type as a fail. That is a coding fault with a known blast radius, and the fix is to re-run the rule across the whole pool with the default corrected. Three are studies reporting stability at twelve months where the abstract mentions only the twelve-month figure, and the scorer had been looking for the phrase "six months".

The seventh contested row is a case the analyst reads and rejects. The advocate argued the program was comparable to those in scope; the review's criteria are explicit that the population is different. The cut stands, and the reason is written on the record.

The failure that ruins this check is a persona that argues too well. Told to defend a candidate, a capable model will find a reading of the evidence that helps, and if nothing constrains it to the record, it will supply the missing piece from general knowledge. Every drop then becomes contested, the analyst stops reading the output, and the check quietly dies.

Guard against it in two places. Require a quoted span from the candidate record for every claim in the defense. Then run the criteria check with the defense stripped of its rhetoric, so the second pass sees claims and citations rather than a persuasive paragraph.

Watch for advocacy that migrates into scope. The strongest case for many rejected candidates is that the round is asking the wrong question. That may be true, and it belongs in a pre-mortem on how the round could produce a useless answer, not in a triage audit. This check tests the cut against the criteria as they stand.

The last failure is scale creep. Once the audit finds something, the instinct is to run it over the whole drop pile every round. That rebuilds the cost triage was meant to avoid and puts an advocacy pass in the critical path. The sample is the method.

Reinstatement is a person's call, made in writing, on a named record. The model produces contested rows. Someone with the round's context reads the case, opens the candidate, and decides. A pipeline that auto-reinstates on a successful defense has replaced its criteria with whichever persona argued last.

Read the recovery count carefully. Seven contested rows out of forty sampled is not a seventeen percent error rate on the drop pile, because the sample was stratified deliberately toward the strata most likely to fail. Report the counts per stratum, with the stratum sizes next to them, or the number will be quoted as an error rate by the second meeting.

Route the systemic findings, not just the candidates. Four rows caused by one default on one rule is a filter fix that affects hundreds of records nobody sampled. That fix is the return on this whole exercise, and it is worth more than the four reinstatements.

Do not confuse this with deep review. Arguing with a triage decision on the basis of a record is a different exercise from multi-agent adversarial review of a full assessment, which tests a considered conclusion against falsification. Here the only claim under test is whether a cut was correct against criteria that were already written down.