Methodology compliance + completeness check

This job has one narrow question. Did the finished work do what the team said it would do? Not whether the conclusion is right. Not whether the method was a good choice. Only whether the draft matches the protocol on file.

The output is a two-column list. Protocol requirements on the left, and on the right one of three states. Present, with a pointer to where in the draft it appears. Missing, with nothing to point at. Or deviated, with the protocol text and the draft text side by side.

That third state does most of the work. Reviews deviate from their protocols routinely, and often for good reasons. A search string turns out to be too narrow. An inclusion window gets extended. The problem is never the deviation; it is the deviation nobody wrote down.

The check does not approve deviations. It finds them and puts them in front of a person who can. A tool that silently accepts a changed inclusion criterion has removed the only control that mattered.

A protocol written as prose cannot be checked automatically. Convert it once into discrete, numbered requirements. Each requirement should be a sentence with a verifiable object. "Report the number of records excluded at each screening stage" works. "Follow best practice in screening" does not.

Do this before the deep dive starts, not afterward. A checklist assembled from the finished draft will match the finished draft. Somebody will unconsciously write requirements the work already satisfies. Freeze the checklist with a version identifier and a date.

Reporting standards and internal rubrics both belong on the list. PRISMA items, a pre-registration record, and the unit's own rubric overlap without matching. Keep them as separate columns so a reader can see which authority a requirement comes from. Failing an internal rubric item is a different conversation from failing a published standard.

State what happens when the protocol is silent. The right behavior is to return nothing for that item, marked not covered by protocol. A checker that invents a requirement from general methodological knowledge has become a critic. That is a useful role and a different one.

Source-side commitments deserve their own items. If the protocol promised a targeted sweep of practitioner reports and registries, the check should ask where that sweep is documented. Absent grey literature is one of the most common silent deviations.

Completeness and compliance are two different passes. Run them separately and report them separately. Completeness asks whether a required section exists at all. Compliance asks whether the section that exists does what the protocol said.

Completeness is mostly structural and cheap to run. Does the draft have a flow diagram, a search date, a list of excluded studies with reasons? Point at the location or mark it missing. Do not accept a heading with no content underneath it.

Compliance is harder because it requires reading two texts against each other. The protocol says three databases; the methods section names two. The protocol says two independent screeners; the draft describes one screener with spot checks. Both are deviations, and both need the protocol wording quoted next to the draft wording.

Then look for the justification. A deviation with a written reason in the draft is compliant reporting. Whether the reason is good is not this check's business. A deviation with no acknowledgment anywhere is the flag that matters.

Numbers deserve a separate arithmetic pass. Records screened, excluded, and included should reconcile across the text, the diagram, and any table. They frequently do not, because drafts are edited in pieces. Report the mismatch with all three figures rather than choosing one.

This example is invented, including the counts. An education evidence unit has completed a systematic review of school meal programs and attendance. The protocol was registered before screening began. Adaeze Nwosu is the methods reviewer who signs the final version.

The checklist holds forty-one items across three authorities. Twenty-seven come from the reporting standard, nine from the registration record, five from the unit's rubric. The check runs against the full draft, the appendices, and the supplementary files together.

Thirty-three items come back present with locations. Four come back missing. The list of studies excluded at full text has reasons for some entries and blanks for others. The risk-of-bias assessment covers eleven of the fourteen included studies. Two items are simply absent: a funding statement and a protocol amendment log.

Four come back as deviations, and they are not equal. One database in the registration was never searched, and nothing in the draft mentions it. The date window ends four months earlier than registered, with no note. Screening was done by one reviewer with a partial second check. The protocol said two independent screeners.

The fourth deviation is already explained in the draft, and it is fine. Two languages were dropped after translation capacity fell through, and the limitations section says so. It appears on the list, marked justified, and Adaeze reads it once and moves on. That is the correct outcome for a documented change.

The most dangerous result is a fully compliant draft that answers the wrong question. Compliance measures conformance to a plan. It says nothing about whether the plan was sensible. A review can meet every reporting item and still rest on a search string that missed the field.

Keyword matching produces false passes constantly. A checker looking for the phrase will find the heading and mark the item present. Whether the assessment was actually performed on every study is a different question. Require the check to point at the substance, not the label.

Long drafts break the check in a quieter way. Requirements are satisfied in appendices, footnotes, and supplementary tables. A check run on the main text alone reports missing items that exist. False alarms are expensive because reviewers stop reading the list after the third one.

There is also a temptation to use the check as a fixer. Asking the model to write the missing funding statement is fine. Asking it to write the missing risk-of-bias assessment is fabrication with a compliance badge on it. Flag the gap; do not fill it.

Compliance never substitutes for substantive challenge. A pre-mortem asking how this round produced a useless answer catches design problems this check cannot see. So does a review by skeptical methodologist and domain personas, which argues with the conclusions rather than the format.

Every deviation ends with a person writing a sentence. Either the deviation is acknowledged in the draft with a reason, or the work is corrected. No third option exists. A flag that stays open when the review is published is a flag nobody owned.

The methods reviewer decides which gaps block release. A missing funding statement is a fix before publication. A missed database may mean re-running the search, or it may mean a limitations paragraph. That judgment depends on what the review will be used for.

Keep the resolved checklist with the published output. Six months later, a reader who asks why one database is absent should find the answer attached. Reconstructing that reason from memory produces a worse answer every time.

Deviations should reach the audience, not stop at the methods file. When the work is repackaged, caveats have a way of falling off the executive version. That is why briefs written per audience with unhideable caveats matter more for a review with known deviations than for a clean one.

One last boundary is worth naming. This check confirms the draft says what it should say. It does not confirm that the cited sources support the claims they are attached to. That is the job of a verifier matching each claim to its cited passage, and both checks are needed.