Bar-setter from prior decisions

Program officers and review chairs hit the same problem at the start of a round. The written criteria say one thing, and the last four rounds did something else. This job pulls decided cases out of the internal archive: what was proposed, what the panel chose, what it wrote at the time, and what happened afterward. The result is a shelf of comparables, not a threshold.

The implicit bar lives in the gap between the rubric and the record. A rubric can ask for methodological rigor while the archive shows proposals with near identical methods sections split on something nobody wrote down. A named partner site. A prior completed award. Those are the operating gates, and they appear only in the decisions.

A case row is usable when it carries the round, the decision, the panel's recorded reasons, and a location a colleague can open. The outcome field stays empty where no outcome exists yet, and a blank is the honest entry for a recent decision.

This is not the same job as pulling prior internal work with the decision attached. That retrieval answers what the organization knows about the subject in front of it. This one answers how the organization has judged things like it.

Three artifacts have to exist before this runs. The submitted item, whatever it is: proposal, protocol, candidate, business case. The decision record, meaning what was chosen, in which round, on what date, with the reasons as written at the time. And the outcome record, meaning what happened over whatever horizon the organization tracks.

Most archives hold the first and lose the other two. Decisions end up in meeting minutes, in a chair's mailbox, or in a spreadsheet tab that was renamed twice. Outcomes end up in a final report nobody indexed back to the decision that funded it. Reconnecting those three is the real effort here, and it is records work more than modeling.

Two further fields matter. The criteria version in force at the time, because bars move and a decision made three years ago was made against the rubric of three years ago. And panel composition, at least at the level of which standing members sat.

Store rejection reasons in the panel's own words with a locator. A reason retyped into a category picklist a year later is not evidence of what the panel thought.

If the archive holds applications and no decisions, return nothing. Similarity search over past applications is retrieval. It says nothing about the bar.

The informative cases sit on the boundary. Ranking archive items by similarity to the proposal in front of you returns the obvious matches, and obvious matches were decided for obvious reasons. Pull three groups instead: the weakest item that was approved, the strongest item that was declined, and any case where the decision went against the recorded reviewer scores.

Then pair them. Two cases that score within noise of each other on the written rubric, one approved and one not, hold the implicit criterion between them. Read the panel text on both. The difference is usually a single unwritten condition, and it is usually specific enough to quote.

Quote it. Keep reasons in the panel's language with a pointer to the minute or the scoring form. Summaries drop the hedge, and the hedge is often where the real gate was.

Do not average scores into a threshold. A mean of 3.6 across four rounds describes the people who sat on those panels. It is not a standard the organization holds, and publishing it as one makes it one.

Order the cases by round so drift is visible. A criterion that fired in the first three rounds and then stopped is a different finding from one that has held throughout. Where the reasons were never recorded, label the row unexplained and leave it in the set. Missing reasons are part of what the archive shows.

This walkthrough is hypothetical, with invented numbers. A biomedical funder runs a translational awards program twice a year. Seven rounds sit in the archive, roughly 40 proposals each, about 6 funded per round.

Retrieval returns 42 decided cases in the current call's therapeutic area and award band. Twenty-nine were declined and thirteen funded. Outcome reports exist for 19 of the 42, because the three most recent rounds have not reported yet. Those remaining rows carry an empty outcome field.

The boundary pairs are the useful part. Three funded proposals scored between 3.1 and 3.4 on the five-point rubric. Four declined proposals scored between 3.5 and 3.8. On the written instrument, the declines scored higher. Reading the panel text on those seven cases, each funded proposal named a clinical partner site and attached a signed letter. Three of the four declines described the partnership as under discussion.

Two reversals point the same way. In both, the panel funded against the reviewer recommendation, and in both the lead investigator had already completed an award recorded in the same archive.

On the outcome side, eight of the thirteen funded projects have reports. Two closed short of planned enrollment, and both had added their partner site after the award was made.

The program officer now has one sentence with seven cited cases behind it: a signed partner letter is doing more work than the rubric admits. That is either a criterion to write into the next call or an institutional habit worth arguing about in the panel meeting. It is not a finding that letters double anyone's odds. Forty-two cases in one program cannot carry that claim, and the report should say so on the row.

Reconstruction fails in ways that all produce a confident looking pattern.

Thin history is the first. Seven rounds is a handful of decisions per criterion once split by area and award size. Treat each pattern as a lead, and print the case count beside it.

Criteria drift is the second. If the rubric was rewritten between round three and round four, cases either side of that line are not comparable, and a pattern spanning it may be an artifact of the rewrite. Record the criteria version per row.

Survivorship is the third. Archives keep approved work well and declined work badly. Declined items get purged on a retention schedule, and their reasons were thinner to begin with. A bar reconstructed only from approvals describes what was approved, not what the panel refused.

The fourth is the one people miss. A faithful reconstruction reproduces whatever habit the organization already has. If the same three institutions kept winning, the reconstructed bar encodes whatever those institutions had. Diagnosis is not endorsement, and the output should never be phrased as a rule.

Anchoring is the last. Showing a panel the historical bar before they have read the current proposals turns a diagnostic into an instruction. Run it before the call opens, or after scoring, not in the room during it.

Reviewer identity stays out. Where confidentiality applies, quote the reasons and drop the names. Rater level patterns belong with tracking scores against outcomes over rounds, under whatever access rule already covers that work.

The reconstruction is evidence about the organization's own behavior. Whether that behavior should continue is a policy decision with a named owner, usually the panel chair or the program lead.

Three responses are honest, and the chair picks one. Write the criterion into the published criteria, so applicants can see the gate. Argue it down as a bias and instruct the panel against it. Or leave it informal and record that it was left informal, which at least means the next round knows what it inherited.

Do not wire the output into a screen. A reconstructed bar has no authority to knock anything out, and a gate nobody agreed to should not quietly filter the pipeline. If it does become a screen, it becomes one after the chair writes it as a criterion and it goes through the same critique any new criterion gets, such as a pass that hunts for gameable and overlapping criteria.

Write the ruling down where the next round will find it. Store the surfaced criterion, the cases behind it, the chair's decision, and the date, in the cross-round store the team can search. This work compounds only when this round's ruling becomes next round's archive entry.