Implementation feedback loop

An analyst hands over a review and then hears nothing. Months later somebody mentions in passing that the main recommendation did not survive contact with the production line. That gap is what this workflow closes, and a survey does not close it.

What comes back is a verdict per claim. The receiving team is asked, on a schedule, which specific claims they tested, what happened, and which ones they could not test at all. Three outcomes per claim: held, did not hold, not yet exercised. A comment box asking how useful the research was returns nothing anyone can fold back into the evidence base.

Verdicts need the same evidence discipline as the original review. "Claim 9 was wrong" is not usable. "Claim 9 predicted cure at 24 hours; we saw 31 to 34 hours across four runs at the line's actual ambient temperature" is usable, because a later reader can see what changed. The recipient is running an experiment whether they call it that or not. Capture it as one.

This is expensive work. Somebody has to chase the check-ins, read the replies, and write the amendments. Teams that want the benefit without the chasing end up with a mailing list.

Not every claim can be tested by the receiving team. Sort them at hand-over, in the room, while both sides are present.

Three buckets. Claims the recipient will exercise directly while doing the work. Claims that would only surface under conditions this team will never meet. Claims that were never testable by anyone here, because they concern a market, a population, or a counterfactual.

Only the first bucket enters the loop. Including all of them produces a form full of "not applicable" and a recipient who stops replying by the second round.

For each claim in that bucket, write the observable. What does the team see if the claim holds, and what do they see if it does not? Write it as something a process engineer could record without interpreting the research. If no observable can be written, the claim is not in bucket one, whatever it looked like in the meeting.

Record who answers, by name. A distribution list produces a reply from whoever feels guilty first, and that person is rarely the one who ran the build.

Set the failure behavior last. An unanswered check-in produces a blank verdict, not an assumed one. Silence is not confirmation. A claim nobody reported on stays at "not yet exercised" until a person says otherwise.

Quarterly check-ins are convenient for the research team and useless to everybody else. Schedule against events on the recipient's plan: first pilot build, first full run, first shipment, first field return. Those are the moments when somebody actually knows something new.

Write the schedule at hand-over and put the dates in the register. Two or three touchpoints tied to real milestones beat six calendar reminders. If a milestone slips, the check-in slips with it. Asking for results before the build ran teaches the recipient that the loop is paperwork.

Keep each round short. The claims due at this milestone, the observable, and three options. Ten minutes of a busy engineer's time is a realistic budget.

A model can prepare the round: pull the claims due, restate each observable in the recipient's own vocabulary, and draft the questions. It should not interpret the reply. The engineer writes what they saw. The analyst decides what it means for the claim. Where the round registered predictions, as in a scored tournament on 12 and 24 month outcomes, align the check-in dates with the resolution dates so one conversation settles both.

Illustrative walkthrough with invented numbers. A materials research group has reviewed thermal interface adhesives for a new power module and recommended a two-part epoxy over the incumbent tape. The review carries 18 claims. At hand-over, manufacturing engineering sorts them: 6 are testable on their line, 5 need automotive thermal cycling they will never run, and 7 concern supplier economics they cannot observe.

Six claims enter the loop with observables attached. Claim 4: cure completes within 24 hours at line ambient. Claim 11: rework rate does not exceed the tape baseline. Claim 15: no additional fixturing required.

Check-in one lands at the first pilot build of 200 units. Claim 15 does not hold. The line needs a clamping fixture nobody mentioned, because the review's sources tested flat coupons and this module has a slight bow. Claim 4 comes back partially held: 31 to 34 hours across four runs, on a floor that sits several degrees cooler than the datasheet assumption. Claim 11 comes back blank, because 200 units produced two rework events and the engineer declines to call that a rate.

Check-in two lands at the first full run. Claim 11 now holds. Claim 4 stays at 31 to 34 hours, consistently, and the team has rebuilt the takt plan around it.

Three amendments follow. Claim 15 gains the flat-coupon condition and the bow exception, cited to the pilot build. Claim 4 keeps its original range and gains a second row recording the observed value at a named ambient. Claim 11 is marked held, with the run size attached. The 12 claims outside the loop stay untouched, and the register says so.

The first round almost always gets answers. People are still warm on the project. The second round is the real test.

Recipients stop replying when the loop costs more than it returns. If the analyst asks and never sends anything back, the check-in is pure extraction. Close the loop in both directions: after each round, send the recipient the amended claims, including the ones their own report changed. Seeing their observation written into the evidence base is the only reliable reason to answer round three.

Recipients also answer defensively when the loop reads as an audit. If the questions sound like they are grading the implementation, the replies get vague. Ask what happened. Never ask why it was not done as recommended. The research team is testing its own claims, not the recipient's execution.

Attribution creep is the third failure. A project that succeeds gets credited to the research and a project that fails gets blamed on it, and neither claim survives inspection. Keep verdicts at claim level. One claim not holding is a fact about that claim.

The last failure is silent overwriting. Somebody edits the original review to match the field result, and the round's original position disappears. That destroys the record of what the researchers believed at decision time, which is exactly the input a tracker comparing rater scores against outcomes needs to work.

The analyst who owns the review writes the amendment. Not the model, not the recipient. The recipient supplies the observation. Turning an observation into a change in the evidence base is research work and it needs a name on it.

Every amendment carries four fields: the original claim as written, the observation that challenged it, the amended claim, and the date. Append, never overwrite. Six months on, the useful question is usually what the team believed before the field result arrived.

Some verdicts should produce no amendment at all. One pilot build is one pilot build. If a claim held across a dozen sources and failed once in a specific plant, the honest amendment adds a condition rather than reversing the claim. The analyst decides which of the two it is, and writes the reasoning down beside it.

Amended claims have to be findable by the next team, which means they belong with the round they came from. Filing verdicts into a searchable store across rounds is what lets prior work retrieved with what happened next surface them. A verdict living in an email thread does not exist.

Repeated challenges to the same claim are a different signal. When two or three recipients report the same failure, the claim is probably wrong rather than conditional, and the review needs re-running instead of annotating. That handoff belongs with periodic rechecking of load-bearing claims.