Claim-grounded source verifier

An analyst doing a first pass on a candidate study has one practical question: do the things this paper asserts appear in the things it cites. The output of this job answers that and nothing else. It is a ledger with one row per checked claim, carrying the claim as written, the citation as given, the passage located in the cited source, and a verdict on whether that passage carries the claim.

A row without a quoted passage and a locator is not a check. It is the model agreeing with the author. Any row that cannot show the sentence it read stays unresolved and goes to a person.

The ledger does not say whether the claim is true. A well-supported claim can rest on a weak study, and a badly cited claim can still be correct. Citation grounding tests one link in the chain: whether the author's sentence and the cited text say the same thing. Design, analysis, and protocol adherence belong to the completeness and protocol check that runs in deep assessment.

It also does not rank candidates. At this stage the ledger feeds a keep, drop, or read-in-full decision that somebody makes later, alongside whatever else the first pass produced.

Two artifacts have to exist before the first comparison: a claim list and a resolved reference list. Building either one badly ruins everything downstream.

Define which claims are in scope, in writing. A workable default is every sentence in the abstract, results, and conclusion that carries a citation marker, plus any sentence that asserts a magnitude, a direction, or a causal link. Background prose restating the field's common ground can be excluded, but that exclusion should be a written rule, not a mood.

Resolve every citation to an object the checker can open: a DOI, a stable report URL, a page range, a dataset identifier. A reference string alone is not enough. Two papers by the same group in the same year are a routine cause of the wrong document being checked against the right claim.

Record access status per reference before the run: paywalled, retracted, withdrawn, superseded by a journal version, or simply missing. Version and retraction status are cheaper to pull during the sweep of registries, practitioner reports, and retraction notices than to discover halfway through a ledger.

If a claim has no citation, the row still exists. It is marked uncited and the verdict stays empty. Filling that cell with a source the model found on its own turns the job into a literature search and hides the original gap.

Score each claim against the located passage. Keep the verdict vocabulary small enough that two reviewers would use it the same way.

Supported means the passage states the claim, including its population, direction, and any quantity. Narrower than claimed means the passage supports part of it: the right direction in a different population, a single site behind a general statement, an association behind a causal verb. Not supported means the passage is on topic and does not carry the claim. Unretrievable means nobody could open the source, which is a status rather than a result.

Every verdict other than unretrievable carries a quoted span and a locator: page, section, table, or figure. The span is the cited author's words, not a summary of them. Paraphrase is where the two texts quietly merge.

Record the mismatch type as well as the verdict, because the type is what a reviewer acts on.

• Population swap: the passage studied a different group than the claim describes.
• Hedge removal: the source says may be associated, the claim says reduces.
• Quantity drift: a figure rounded, converted, annualized, or read off a different column.
• Chained citation: the cited work is itself citing someone else for this point.
• Scope inflation: a result from one setting written as a general finding.

Chained citations deserve their own count. Following a single hop is enough for a first pass to show whether a candidate's evidence base is thinner than its reference list suggests.

This walkthrough is hypothetical. An evidence synthesis analyst at a schools funder is running first-pass reads on candidate studies about small-group tutoring. One preprint argues that short daily sessions outperform longer weekly ones across primary and secondary grades.

Claim extraction returns 34 in-scope claims against 41 distinct references. Reference resolution finds 38 openable, 2 paywalled beyond the funder's access, and 1 preprint that now has a journal version reporting different results.

The ledger comes back with 21 rows supported. Nine are narrower than claimed, and six of those nine cite primary-school samples under a sentence that covers primary and secondary. That pattern is the finding worth carrying forward, because the preprint's central comparison rests on it.

Two rows are not supported by the cited passage. One cites a review that discusses session frequency and never compares frequency with duration. The other attaches a specific figure, 40 minutes, to a source whose tables report 45 and 30 with no combined value.

Two rows stay unretrievable. Three are chained: the cited paper is quoting an older evaluation, and that evaluation is what would have to be read.

The analyst keeps the preprint in the pool, writes on the row that generalization to secondary grades is not supported by its own citations, and lists the three primary sources to open if it moves forward. What nobody does is reject the preprint because two of 34 citations missed. Two scattered misses are ordinary. Six misses pointing the same direction on the load-bearing comparison is not.

The check fails quietly in a few predictable ways, and each one produces output that looks complete.

Abstract-only matching is the most common. When only the abstract is reachable, the model finds agreeable sentences there and marks rows supported. Abstracts overstate. Record which rows were checked against full text and which against an abstract, and treat the second set as unverified.

Semantic near-matches pass when they should not. A passage on the same topic, with the same vocabulary and a different population, scores high on similarity and low on truth. Require population, outcome, and direction to match explicitly, not the overall subject.

Numbers slide. A percentage in a claim can appear in the source on a different base, over a different window, or in a subgroup. Any claim carrying a number gets checked digit by digit against a table or a stated figure, never against surrounding prose.

Volume drowns the reviewer. A hundred candidates times thirty claims is a queue nobody works. Cap the run at claims that could change the keep-or-drop call, and merge near-identical candidates first through the duplicate and alias merge that runs during triage.

The last failure is over-reading a single miss. One broken citation in a long paper is ordinary editorial slippage. Report counts, patterns, and where the misses cluster, not a pass or fail stamp.

A verdict on a row is not a decision about a paper. The analyst reads the ledger and decides what the pattern means: keep, drop, or promote to a full read with named sources to check. That judgment depends on where the misses fall, and only a person knows which claims the argument actually rests on.

Escalation is a human act. A pattern that looks like misrepresentation rather than sloppiness goes to a named senior reviewer through whatever process the institution already uses. The ledger is an input to that conversation. It is not an accusation, and it does not go to an author as one.

Blanks stay blank. Unretrievable rows are not failures, and they should not become supported rows once somebody finds a similar paper. Record access as the blocker, so a later reader knows the check was incomplete rather than clean.

Carry the ledger forward. Verified and flagged claims should travel with the candidate into the grid whose cells are filled from each source, so a cell built on an unsupported claim is visible next to the claim itself. If the candidate later underpins a published position, the flagged rows are the first thing to re-run when the refresh trigger checks whether conclusions have gone stale.

Name who checked what. A ledger with no reviewer on it is an artifact people cite for months without knowing whether a human ever opened a source.