Living-document refresh trigger

Evidence reviews do not go stale evenly. Most of a fifty-page review stays true for years. A handful of claims sit under the conclusion and hold it up, and when one of those moves, the conclusion moves with it. This workflow watches that handful.

The output is a flag on a specific claim with the new evidence attached: this claim, this source, published on this date, and here is the passage that says something the review's sources did not. A monthly digest of everything published in the field is a different product and a worse one.

Re-running the whole review on a schedule is the alternative most teams reach for, and it is why so few of them keep it up. A full re-run costs nearly as much as the original round. Repeated annually across a portfolio of reviews, it consumes the team that was supposed to be answering new questions.

This is expensive to operate and the payoff is thin in most quarters. It earns its place on reviews that are still being cited to make decisions, and only those. A review nobody reads does not need a monitoring budget.

Do this at hand-over, while the reasoning is still fresh. Later nobody remembers which claim was carrying the argument.

For each candidate claim, ask one question. If this turned out to be wrong, would the recommendation change? Most claims fail that test. They are context, background, or supporting detail. The ones that pass are few. A review with more than about a dozen load-bearing claims has probably not been analyzed properly yet.

Write a movement threshold for each survivor. Not "watch for new evidence." Something checkable: a new trial with a sample above a stated size, a regulatory decision in a named jurisdiction, a cost figure outside a stated band, a retraction of a source the review relied on. The threshold is what separates a trigger from a reading list.

Record what counts as the opposite too. Confirming evidence matters, because a claim that has survived four new studies is stronger than it was, and the review should be able to say so.

If a claim has no writable threshold, it stays off the watch list. A claim about whether a program is politically survivable may be load-bearing, but nothing will trip on it automatically, and pretending otherwise fills the queue with judgment calls the system cannot make. Those belong to a scheduled human read.

Keyword alerts on the topic produce volume. The conclusion does not care about volume. It cares whether a specific condition still holds.

Set the watch on three things. The sources the review actually relied on, so a correction, retraction, or revised version reaches the claim it supports. The registries and outlets where a contradicting result would appear, which is a narrow list for most questions. And the assumptions the review imported from outside its own evidence: a price, a rule, a population estimate the argument silently depends on.

Collection is the easy part, and a continuous watch on defined sources can run it. The hard part is the second pass, deciding whether an incoming item speaks to a named claim at all. That check has to be claim-specific. Ask the model to identify the passage in the new source that addresses the claim, and to return nothing when it cannot find one. An item with no locatable passage is not a trigger.

Breadth of retrieval belongs to semantic search over the right corpus, not to this step. Refresh monitoring is deliberately narrow. Widening it to catch everything relevant is how the queue becomes unreadable inside two months.

Hypothetical scenario with illustrative numbers. A district research office published a review eighteen months ago on high-dosage tutoring, concluding the format was worth funding at a stated ratio and dosage. Schools are still budgeting against it, so it stays on the watch list.

Seven claims are load-bearing. Two carry most of the weight. Claim 2 holds that effects at a tutor-to-student ratio of one to three are close to those at one to one. Claim 5 holds that gains persist at least one school year after the program ends.

The thresholds were written at hand-over. For claim 2: any randomized study above 1,000 students that reports outcomes by ratio. For claim 5: any study reporting follow-up beyond twelve months, in either direction.

Over eighteen months the monitor surfaces 34 items. Twenty-six are screened out at the passage check, because they discuss tutoring without reporting by ratio or beyond twelve months. Six are confirming, sitting under claim 2 and strengthening it, and the register records that in a batch. Two trip.

The first is a large trial reporting weaker results at one to three than the review's sources found, on a sample of roughly 4,000. It clears the threshold. The claim is flagged stale, not wrong: one trial against the review's four does not reverse a claim, and the flag says so on its face.

The second is a follow-up on a cohort the original review already included, reporting that the gain had faded by month eighteen. It sits directly under claim 5, and it comes from a source the review relied on.

Both flags reach the analyst who owns the review. Claim 5 turns out to be the one that matters, because the district's funding case assumed persistence. The review reopens for those two claims. The other five, and the forty pages around them, are not touched.

The failure that kills refresh monitoring is not a missed paper. It is forty flags in a month, none of which changed anything, and a research office that stops opening the digest.

Loose thresholds cause most of it. "New evidence on tutoring" fires constantly. "A randomized study above a stated sample reporting by ratio" fires rarely, and every firing is worth reading. Tighten thresholds until the queue is small enough that somebody reads all of it.

Confirming evidence gets mishandled in both directions. Some setups ignore it, so the register never records that a claim has strengthened. Others treat every supporting paper as an event, which is the same flood under another name. Count confirmations, report them in batches, and flag only contradictions.

Preprints and single studies belong in a lower tier. A result that has not been reviewed, or has not been replicated, can mark a claim as worth watching without marking it stale. Two tiers are enough.

The quiet failure runs the other way. Somebody configures the monitor, nobody is named as the reader, and eight months later the queue holds ninety unread items while the review still says what it said. An unread queue is worse than no queue, because the team believes it is covered.

A trigger says a load-bearing claim has new evidence against it. It does not say the conclusion has changed, and nothing downstream should read it that way.

The analyst who owns the review makes that call, and it has three possible endings. The claim holds, and the new evidence is an outlier or out of scope, in which case the register stores the item and the reasoning. The claim narrows, gaining a condition it did not carry before. Or the claim fails, and the review reopens.

Reopening is scoped, not automatic. Two claims out of seven can be re-run for a fraction of the original effort. Reopening everything because one claim moved is how maintenance turns into a permanent research project.

Where confidence was recorded as a distribution, as in a register of decisions held as probabilities, the update is often a shift rather than a reversal. Say so plainly. Moving a claim from strong to contested is a real result and it reaches readers faster than a rewrite.

Downstream artifacts go stale with the claim. When a claim changes, every version built from it is out of date, so the briefs written for each audience get reissued together. Teams already acting on the review need telling directly, through whatever structured check-ins with the recipient are running. A stale flag that never reaches the person who budgeted against the claim has done no work at all.