Calibration tracker

Research funders and review leads arrive at this question after a few rounds: did the people scoring proposals predict anything? The output of a calibration tracker is a per-rater history. Each line holds the round, the proposal identifier, the criterion, the score that rater submitted, the score the panel settled on, and the resolved outcome. Nothing is averaged before a person asks for an average.

That artifact is not a ranking of reviewers. It is not a performance file, and it should never reach a panel member's employer as one. A rater who runs a point above the panel on methodology is telling you something about the scale, the criterion wording, or their reading of the evidence. Which of those it is takes a conversation.

The distinction that matters most is between agreement and accuracy. Agreement compares a rater to the rest of the panel. Accuracy compares a rater to what happened. A panel can agree closely and be wrong together. Keep those two columns apart, permanently.

Patterns of this kind sit next to what past funding decisions reveal about the implicit bar. That work asks what the organization rewarded. Calibration asks who in the room saw it coming.

Calibration needs an outcome, and the outcome has to be written before the round it will judge. Four inputs get loaded. The resolution rule: what counts as the funded work having succeeded, stated as something checkable on a date. The resolution date itself. The criterion map, so that methodological rigor in round one means what it means in round four. And the raw individual scores as submitted, before panel discussion moved anyone.

That last input is the one teams lose. Many panels record only the agreed final score. Once discussion has run, every rater's number converges, and the record shows a panel agreeing with itself. Capture the pre-discussion score with a timestamp, or the tracker has nothing to work with.

Outcome definitions have to survive the awkward cases. "Delivered" is not a rule. "Reported the pre-registered primary outcome to the funder within 90 days of the stated end date" is. Write the edge cases in now: approved extensions, a principal investigator who moves institution, a study stopped early for a defensible reason.

If no outcome is defined, or the criteria were rewritten between rounds without a version tag, run nothing. A tracker that maps an old rubric onto a new one reports drift that is an artifact of the edit. Criteria built to be judged independently, in the way scoring each sub-criterion on its own before recombining describes, give the tracker something stable to follow.

Three comparisons, three columns, never one blended calibration score.

Agreement is the rater's submitted score against the panel's central score on the same proposal and criterion. Report signed direction and size. "Two points above the panel on impact, eleven times out of fourteen" is a pattern a chair can act on. "Low agreement" is not.

Accuracy is the rater's score against the resolved outcome, on proposals where an outcome exists. It has two directions and they need separate columns. Over-rating means scored high, outcome poor. Under-rating means scored low, outcome good.

Drift is the rater against their own earlier rounds. Scale use compresses over time. A reviewer who used the full range in the first round and nothing but threes and fours by the fourth has changed instrument, and the panel's spread narrows along with them.

Report counts rather than rates while the resolved sample is small, and print the count on every line. Under a couple of dozen resolved proposals, a percentage invents precision the data does not have.

Keep everything per criterion. A reviewer can be sharp on feasibility and unhelpful on novelty, and blending those into one number destroys the only finding worth having. Variance inside a live round is a different job, handled by flagging high-variance candidates before the vote. Calibration works after.

This walkthrough is hypothetical, with invented names and numbers. A funder running an implementation-research program holds two rounds a year. Across four rounds it scored 214 proposals and made 61 awards. Three criteria are scored 1 to 5: methodological rigor, feasibility, and policy relevance. The resolution rule, registered before the first round, is the reporting deadline above.

Rounds one and two have now resolved, giving 29 awards with outcomes.

Devi Raman, a methodologist who sat on all four panels, scored rigor a median of 0.8 points below the panel across 71 proposals she reviewed. Of the 29 resolved awards, 9 missed the reporting rule. Raman had scored rigor at 2 or below on 7 of those 9. The panel's agreed rigor score on the same 9 was 3 or above in 6 cases.

The tracker prints those as counts and marks the sample thin. Nine resolved failures do not validate a reviewer. They give the panel chair a reason to open Raman's written comments on those seven proposals and find out what she was reading that the room discounted.

A second line runs the other way. Marcus Oyelaran scored policy relevance close to the panel every time, with no visible relationship to the resolved outcome. That is a question about the criterion rather than a finding about Marcus.

A third line is pure drift. One reviewer used all five points of the scale in round one and only two of them by round four. Their contribution to panel spread collapsed.

Selection is the structural problem and it never goes away. Only funded proposals produce outcomes. Nobody learns what the rejected ones would have done. Every accuracy column is conditional on the panel's own decisions, and that caveat belongs on the artifact itself, not in a methods appendix nobody opens.

Proxy outcomes become targets. If on-time reporting is the resolution rule, panels quietly start scoring for administrative reliability, and the criterion that was supposed to measure rigor drifts toward measuring project management.

A visible scoreboard changes the scores. Raters who can see their own agreement number start scoring toward the panel. That is the one behavior a calibration program must not train, because independent reads are why a panel has more than one person on it. Show raters their own line privately.

Rater turnover thins everything. Two rounds is not a history. Say so on the row instead of extrapolating from three proposals.

Attribution is the last trap. An award that failed because a collaborating hospital withdrew was not badly designed. Outcome data carries causes the score never claimed to predict.

A named person owns every response to this record. In most funders that is the panel chair or the research director who convenes the panel. The tracker assembles evidence about how scores related to outcomes. It does not decide who reviews the next round.

Automatic weighting is the failure to refuse first. Multiplying a rater's score by a coefficient derived from past accuracy hands the decision to a model and makes the next round's numbers uninterpretable. If the chair wants a reviewer's methodology score to count for more, the chair says so in writing, before the round, with a stated reason.

The defensible responses are ordinary and human. Rewrite a criterion that has stopped discriminating. Add a second methodologist where rigor scores keep separating from outcomes. Ask a reviewer to walk through the cases where they dissented. Change nothing and note why.

Record what the chair decided, when it takes effect, and which lines supported it. Filed with the scores in a repository searchable across rounds, that note lets the next chair understand why the rubric reads the way it does.

Keep this separate from asking people to predict on purpose. Registered forecasts on 12- and 24-month outcomes collect a judgment the forecaster knew would be scored. Calibration reuses scores given for another reason, and the record should not treat the two as the same evidence.