Disagreement flagger + pre-meeting brief

Triage meetings run long because the room works through every row at the same pace. Most rows need no discussion at all. Two reviewers scored them the same way for the same reasons, and reading them out loud wastes the hour. A disagreement flagger finds the rows where that is not true.

What it hands over is a list of candidates with a short brief on each. The brief says where the two scores split, which sub-criterion carries the split, and what each side seems to be reading differently. It ends with the question that would settle the matter. Nothing in it recommends an answer.

This is a speed tool. It improves no score and adds no evidence. It moves attention to the four or five rows where the call is genuinely open. On a clean round it may flag almost nothing, and that is a useful result too.

Disagreement also says something about the criteria. When one sub-criterion splits reviewers row after row, the wording is loose rather than the candidates being hard. Testing criteria for overlap and gameability belongs back at scope, but triage is where weak wording shows up.

The flagger needs at least two scores per candidate, produced without sight of each other. Two human reviewers work. One human and one model work. A single model run scored twice does not, because both runs share whatever made the first one wrong.

Scores have to sit at the sub-criterion level. A total is an average, and averages hide the split that matters. Two reviewers can both land on 14 out of 20 while disagreeing sharply about cost and about the strength of the evidence. Scoring each sub-criterion on its own is what makes this step possible.

Set the threshold before the round opens. State how far apart two scores must be on a sub-criterion before the row is flagged. On a five-point scale, a gap of two points is a common line. Write it down. A threshold picked after seeing the spread just chooses which arguments the room will have.

A missing score is not a disagreement. If one reviewer never scored a row, that row goes to a chase list rather than to the meeting. Treating a blank as a zero creates a flag with nothing behind it, and the room works that out in the first minute.

The mechanism is plain arithmetic followed by one drafting step. Compute the spread per sub-criterion for every candidate. Keep the rows where any sub-criterion clears the threshold. Order those rows by the size of the largest gap, not by the total.

Drafting is where a model earns its keep. For each flagged row it reads both sets of written reasons and states the split in a sentence or two. One reviewer called the evidence weak because the only trial ran on a single site. The other called it strong because the effect held over three seasons.

Then it writes the resolving question. A good one can be answered in the meeting or by one named piece of work. The brief asks whether the protocol accepts a single-site trial as sufficient evidence here. That question closes. Which reviewer has better taste does not.

The brief quotes reasons instead of summarizing them. A summary in the model's words gives both reviewers something to object to before the real argument starts. Where a reviewer left the reason field empty, the brief says so plainly. An empty reason is worth seeing on the page.

The round below is a construction, with counts chosen only to show how the step plays out. A state forestry agency is screening candidate fuel-treatment methods for fire-prone slopes. Sixty-two candidates survive the earlier filters. Two silviculturists score each one on four sub-criteria: strength of evidence, fit to steep terrain, cost per hectare, and crew readiness for next season.

The threshold is a two-point gap on a five-point scale. Nineteen rows clear it somewhere. Eleven of those split on crew readiness alone, and their briefs read almost the same. One reviewer counted a method ready when a written procedure existed. The other wanted trained crews already in the district.

That is one argument, not eleven. The chair lifts it out as a definition question and puts it first on the agenda. Ten minutes settles it, and the eleven rows are rescored against the agreed reading with no further debate.

Eight rows remain. Four split on evidence and four on cost. The cost splits turn out to be arithmetic: the two reviewers used different denominators, one per treated hectare and one per hectare that stayed treated after three years. The four evidence splits are the real work, and the meeting spends most of its hour there.

Sixty-two rows became four arguments, and the briefs decided none of them. Re-scoring a sample of dropped candidates with an advocate persona covers the opposite risk, which is a room that agreed too easily on a row worth arguing about.

What kills this practice is a pack full of flags nobody needed. Every false flag spends the same five minutes as a real one. After two rounds of that, the room stops reading the pack at all.

Loose thresholds cause most of it. A one-point gap on a five-point scale is ordinary rater variation, and flagging it produces a list as long as the shortlist. Raise the threshold until the flagged set is short enough to work through properly.

One rater's habits create the second kind of noise. A reviewer who marks everything a point low will differ from everyone, on every row, in the same direction. That is a calibration matter, and tracking each rater's scores against outcomes over rounds is where it gets addressed. A flagger blind to it will simply report the same person nineteen times.

Briefs that take a side do real damage. A model asked which reviewer is more likely right will answer, and it will sound sure. Reviewers then argue with the brief instead of with each other. The brief describes the split and stops there.

The flagger proposes an order. The person chairing the session decides what the room actually discusses. Some flagged rows can be closed by email. Some unflagged rows still need airtime because a reviewer asked for it. An agenda that cannot be overridden is worse than none.

Every flagged row needs a recorded ending. Agreed on the higher score, agreed on the lower, or held for a named piece of work with an owner and a date. A row that leaves the meeting still split has not been triaged. It will come back at the decide stage with less time around it.

Reasons matter more than the final number. When a split is settled, someone writes the why in a line or two. That line is the raw material for the record of which assumption each side rested on, and it costs almost nothing to capture on the day.

A flag says nothing about candidate quality. It says two people read the same row differently. Weak candidates often draw perfect agreement, and the strongest entry in a round can be the one that split the room hardest. Only the reviewers decide what the disagreement meant.