Comparison matrix auto-fill

Researchers reach for a comparison matrix when the candidate set has grown past the point where anyone can hold the differences in their head. The columns are the questions the review cares about. The rows are the candidates. Auto-fill takes the reading and typing out of the middle: the model reads each source and writes what it says in every cell.

That is the whole promise, and it is a modest one. The grid moves facts out of documents into one shape. It does not decide which candidate is stronger, and a filled row is not an assessment. The comparison happens when a person reads down a column and notices that four studies define the exposure differently.

It is cheap to stand up, cheap to run, and easy to overrate. A matrix looks authoritative long before anybody has checked a single cell, and that gap is the main thing to manage.

Fill from sources, never from background knowledge. If a candidate never states its funding source, the funding cell is empty. A model that knows the usual funders in a field will produce a plausible cell no source supports.

The columns are the design work, and the quality lives there. Define each one before the first source is read, in language precise enough that two people extracting the same paper would write the same cell.

A column needs four things: what is being asked, the unit or vocabulary the answer must use, what counts as not stated, and an example of a correct cell. "Sample size" is ambiguous across enrolled, analyzed, and followed up. "Number analyzed in the primary outcome model, as an integer" is not.

Keep the column set short enough to fill honestly. Twelve well-defined columns beat thirty that degrade into free text halfway across the grid. Columns that restate the screening criteria are usually the right ones, and those criteria exist already from the translation of the brief into a hypothesis tree and screening criteria.

Distinguish extraction columns from judgment columns, and keep them apart on the sheet. Extraction asks what the source says. Judgment asks what the reviewer thinks. Ratings such as risk of bias or relevance are judgment, and they belong to the independent scoring of each sub-criterion, not to an auto-filled cell.

If the column definitions are not written, do not run. A grid filled against vague columns cannot be repaired later, because nobody can tell whether an odd cell reflects the source or the prompt.

Each filled cell carries the value, a short quoted span, and a locator: page, section, table number, or figure. Without that trio, a wrong cell is undetectable, and checking it means re-reading the whole paper.

Fill candidate by candidate, not column by column. Reading one source once and answering all twelve questions from it keeps the model inside a single document. Sweeping a column across twenty-six sources invites convergence on whatever the majority said.

Use a fixed vocabulary for missing values, and make the categories mean different things. Not stated means the source is silent. Not applicable means the question does not apply to this design. Unclear means the source is ambiguous and a human should read the passage. Collapsing these into one blank destroys the grid's most useful signal, which is where the literature is quiet.

Never normalize silently. If one candidate reports a hazard ratio and another an odds ratio, the cells keep both labels. A model that converts, rounds, or annualizes to tidy a column has created a number that appears in no source. Conversion is a documented analytic step with an owner, done after extraction.

Flag disagreement between a cell and the candidate's own abstract rather than picking a winner. That conflict is worth a reviewer's attention, and it is often the first sign that a claim has drifted from its evidence, which the claim-to-passage check examines closely.

This walkthrough is hypothetical. A research associate at an occupational health institute is doing a first pass on 26 cohort studies of night-shift work and metabolic outcomes.

The grid has eleven columns: cohort name, country, years of follow-up, number analyzed, how shift work was measured, how exposure was categorized, outcome definition, effect measure and value, covariates adjusted for, funding source, and data availability.

Auto-fill returns 286 cells. Of those, 47 come back as not stated and 19 as unclear. The unclear cells cluster in two columns, exposure categorization and covariates, which is itself the finding: those are where the studies are least comparable.

Reading down the exposure column shows four definitions in use. Eight studies count any night work in the past year, six use a lifetime cumulative measure, five use a rotating versus fixed distinction, and seven describe the schedule in prose with no category. Nothing in the grid says which definition is better. The associate now knows the pooled comparison the review planned cannot run across all 26.

Effect measures split as well: 17 hazard ratios, 6 odds ratios, and 3 studies reporting only stratified counts. Those three have empty effect cells, with a note pointing at the table holding the counts.

Spot-checking ten random cells against their locators finds two errors. One pulled the number enrolled rather than the number analyzed. The other lifted a covariate list from a sensitivity analysis instead of the primary model. Both took minutes to catch, because the locator was on the cell.

The failure modes are specific to auto-fill, and none look like errors on the page.

Cells converge toward the majority. When a model sees many similar candidates, ambiguous sources drift toward the pattern the others set. Re-run a sample in shuffled order to see whether any cell changes.

Empty cells get filled to finish the row. A model asked for a complete grid will produce one. Reward the blank in the instruction, and audit the ratio of blanks by column. A column with no blanks across twenty-six diverse sources is suspicious rather than reassuring.

Unit and definition mismatches hide inside neat columns. Currency years, exposure windows, and outcome definitions all differ, and a column of numbers implies they do not. Keep the definition beside the value whenever the column allows more than one.

Extraction errors survive because nobody samples. The grid becomes the reference copy within a day, and the original documents stop being opened. Sample cells for verification on a fixed rule, weighted toward the columns that will drive the decision.

The last risk is scope. A matrix built for a first pass gets reused for a different question six months later, with columns never designed for it. Store it with its column definitions and its date, alongside the round's other artifacts, so a later reader in the searchable repository of decision artifacts can see what the grid was built to answer.

The grid is an input to a person's reading, not a substitute for it. A researcher scans each column for spread, for blanks, and for cells that disagree with their expectation. That scan produces the useful observations, and no auto-fill step does.

Blanks are read, not ignored. A column that is mostly not stated says something about reporting practice in the field, and it may be worth more than any filled cell.

Rows are not scored by counting filled cells. A candidate that reports everything is not stronger than one that reports less, and turning completeness into a rank rewards verbose writing. Ranking runs on defined criteria with a named owner.

Where two reviewers read the same column differently, that divergence goes on the agenda rather than into an average. Surfacing it early is the job of the brief that flags high-variance candidates before a meeting, and a comparison grid is a common place for that variance to appear first.

Record who verified which cells and when. A grid nobody has spot-checked is a set of unverified assertions in a tidy layout, and the layout is exactly what makes people stop asking.