Grey-literature + practitioner sweep

The sweep runs after the published search closes. It produces two things: a short annotated list of non-journal material, and a list of studies that should exist and cannot be found. Both go into the record. Neither one is expected to change the effect estimate.

That is the honest case for running it. Most rounds, a sweep adds caveats rather than findings. It tells the team how much of the field is visible, which is a different question from what the visible part says. When publication pressure plausibly runs in one direction, that ratio matters more than any single new study.

The cost is real and the shape of it surprises people. There is no single index. Practitioner reports live on institutional sites with no metadata. Registries have their own query languages. Correction notices sit in separate databases from the papers they correct. A sweep is several small searches with different rules, and it does not compress into one query.

Keep it separate from the main retrieval pass. Semantic search over a chosen research corpus builds the study set. The sweep tests that set from outside. Running them together buries the distinction between a peer-reviewed trial and a program manual, which is the distinction the whole exercise exists to preserve.

Practitioner and institutional reports are the first list. Evaluation reports, implementation guidance, audits, and internal write-ups from organizations that ran the thing. Read them for what was done, at what scale, at what cost, and what went wrong in operation. Do not read them for effect sizes. Most have no comparison condition and were written by people with a stake in the result.

Registry and protocol records are the second list, and they are read for what exists rather than for what they say. A registration proves a study was planned. Its status field says whether it completed. The gap between registered studies and published ones is the most direct evidence a review can get about what is missing.

Correction and retraction records are the third list, and they run against the candidates already collected rather than adding new ones. Check every study in the set, then check whether any of them cite something that has since been retracted. A retracted paper propagating through three citing studies is the failure this list catches.

Each list has its own retrieval path and its own standard. Dropping all three into one folder labeled grey is how a vendor white paper ends up cited beside a randomized trial in the same sentence.

Every row carries provenance fields before it carries anything else: the producing organization, the funder or sponsor, whether it was externally reviewed, whether the underlying data is available, and the publication date. When the sponsor is not stated, record it as not stated. Do not infer a funder from the logo on the cover.

Archive a copy at retrieval, with the date and the address. Grey material disappears. Agency pages get restructured, organizations rebrand, and a report cited in March is a dead link by the time the review is read. A citation to a document nobody can open is not checkable, and the same rule that governs verifying that a cited passage supports the claim applies here with more urgency, because there is no publisher of record to fall back on.

Registry entries need one extra field: the primary outcome as registered. When a published paper reports a different primary outcome from the one in its registration, that discrepancy belongs on the row. It is not an accusation. It is a fact about two documents, and the deep-assess stage decides what it means.

For the missing list, record the registration identifier, the completion date, the sponsor, and the last status change. A blank reason field stays blank.

This walkthrough is hypothetical and the numbers are invented. A clinical informatics team at a hospital network is assessing a commercial sepsis-alert algorithm before a deployment decision. The published search closed with eighteen evaluation studies in the set.

The registry pass finds nine registered prospective evaluations of alert systems in this class. Four match publications already in the set. Two are marked completed with no results posted and no publication more than three years after their completion date. One is terminated with the reason field empty. Two are still recruiting.

The correction pass runs against the eighteen. One carries a correction notice that restates the alert threshold used, which changes how its results compare with the others. One has been retracted for data handling. Three of the remaining sixteen cite the retracted paper as supporting evidence, and none of them flags it.

The practitioner pass returns six items. Two are vendor white papers reporting case-study numbers with no comparison condition. One is a professional society implementation guidance document. Two are conference abstracts with no full paper behind them. The sixth is a health system quality report describing a deployment that was rolled back after eleven months, with figures on alerts per bed per day and on how nursing staff responded once the rate passed a certain point.

The rolled-back deployment is the item worth the sweep. It says nothing about the algorithm's accuracy. It says what a ward does at that alert rate, which no study in the published set measured.

The effect estimate does not move. What moves is the description of the evidence base: eighteen published evaluations, two completed and unpublished, one terminated without a stated reason, one retracted and still carried by three citing papers.

A vendor report treated as neutral evidence is the first error, and it happens because the report is well produced and reports numbers. Numbers with no comparison group and a sponsor who sells the product are a description of a deployment. Keep them labeled that way through every downstream artifact.

Registry silence treated as proof of suppression is the mirror error. A completed study with no publication is a question. Investigators leave, funding ends, data pipelines fail, organizations merge, and results sit unwritten for reasons that have nothing to do with what they showed. Record the gap, describe it plainly, and do not narrate a motive.

Scoring grey items on the trial rubric breaks both ways. Score a program manual for randomization and it fails out of the review, taking its operational detail with it. Score it as a study and it enters the count as if it were one. Give grey material its own fields and its own place in the write-up.

Running the correction check once, at the start, is the quiet failure. Retractions and expressions of concern arrive late, often after a review is drafted. Re-run the check against the final citation list before the report ships.

The last one is scope. A sweep has no natural end, because there is always another agency site. Time-box it, name the source classes in advance, and stop when the box is empty. If the cull looks too aggressive afterward, re-scoring a sample of dropped candidates with an advocate persona tests that judgment more cheaply than searching for another month.

Two unpublished completed trials do not authorize an adjustment to the estimate. A named methodologist writes what the gap means into the limitations, in words, with the registration identifiers attached. Hand-correcting a pooled result for suspected publication bias produces a number nobody can trace back to a document.

The same person decides what a grey item is allowed to support. An operational report can support a claim about feasibility, staffing, or workload. It cannot support a claim about effect. When a grey item changes a recommendation, name it in the report with its sponsor and its review status visible in the same line, not buried in a bibliography.

Do not upgrade grey material by paraphrase. A sentence that begins "evidence suggests" and rests on a vendor case study has laundered its source. Write which document, produced by whom, describing what.

Whether the sweep was adequate is itself reviewable. Checking the review against its declared protocol should confirm that the source classes named in the plan were actually searched, that the correction check was re-run, and that the unfound studies are listed rather than described in general terms. A sweep with no missing list usually means nobody looked for one.