Criteria stress-tester

Program officers and review chairs arrive at this step with a draft rubric and a nagging feeling. The critique that helps them is a list of specific attack routes. Each entry names one criterion, describes how a candidate could satisfy it without having the underlying quality, and says what evidence would tell the two apart.

The model does not get to replace the rubric. A rewritten set of criteria arrives with no record of what changed or why, and the panel loses the argument that produced the original. Critique first. Revision is a separate act with a named author.

The distinction matters because criteria are political. A rubric is the compressed record of what a group agreed to value. A critique that arrives as a clean replacement erases that agreement and asks the panel to re-litigate it under time pressure. A critique that arrives as a list of specific weaknesses lets them fix three lines and keep the rest.

This is a cheap check on an expensive downstream. Criteria written this week will drive every score in the round. A criterion that two people read differently produces variance later, and that variance shows up as candidates the reviewers cannot agree on.

A critique built on the criteria alone is generic. Three things have to travel with the rubric. The decision the scores will support, because a criterion that works for a funding shortlist fails for a go or no-go on one project. The population being scored, because gameability depends on who is answering. And the evidence each criterion will actually be read from.

That last input does most of the work. A criterion scored from a self-reported application form is open to different attacks than one scored from an audited filing. Say which field, document, or interview the score comes from. If nobody can name the source, the criterion is not yet a criterion.

Load the scoring scale too. A four-point anchored scale and a yes or no gate fail in different ways. Without the scale, the critique will complain about thresholds the team already settled. Run this after the brief has been turned into a hypothesis tree and screening criteria, not before.

When the decision or the evidence source is missing, return nothing. A critique of criteria floating free of a decision reads well and changes nothing.

Run three passes and keep them separate. Blending them produces a paragraph of concerns that nobody can act on.

The gaming pass asks how a candidate maximizes the score without the quality. For a criterion on community partnership, the cheap route is a letter of support signed by an organization that will do nothing. The pass should name that route and name the check that catches it: a named person, a described role, a prior joint output.

The overlap pass looks for criteria that move together. Two rows measuring the same underlying trait double its weight without anyone deciding to. Track record and prior funding usually collapse into one signal. The pass should show which pairs would rise and fall together, and let the panel choose whether that weighting is intended.

The silence pass is the hardest and the most valuable. It asks what a candidate could be terrible at and still score well. Say a rubric covers technical merit, cost, and team. A proposal with no plan for what happens after the grant ends passes every row. The missing dimension is sustainability, and no amount of careful scoring on the other three will surface it.

Each finding should come back in one shape: the criterion, the failure type, one sentence describing the route or the gap, and the proposed fix. Findings written as paragraphs of caution cannot be voted on. Findings written as a row can be accepted, edited, or rejected in a meeting.

This walkthrough is hypothetical. A foundation is opening a round for coastal adaptation projects in small municipalities. The draft rubric has five criteria: local need, technical feasibility, partnership strength, cost per household reached, and organizational capacity. Each is scored one to four by two readers.

The gaming pass flags cost per household first. The denominator is applicant-defined. An applicant can widen the beneficiary boundary to the whole coastline and cut the figure by a large factor without changing the project. The critique proposes fixing the denominator in the guidance: households inside the defined flood zone, using the municipality's own map.

The overlap pass flags partnership strength against organizational capacity. Both are read from the same two application sections. A well-connected applicant scores high twice, so partnership effectively carries two of the five rows. The panel may want that. It should know it is choosing it.

The silence pass returns two gaps. Nothing in the rubric asks whether the municipality can maintain the asset in year six. Nothing asks who bears the risk if the design underperforms. Both would be invisible in every score, and both are the kind of thing the fund's board asks about later.

Two readers then test the revised cost criterion on three applications from last year's round. On one of them, the fixed denominator moves the score by a full point. That single check is worth more than the rest of the critique combined.

The program officer takes four of these forward and drops one. The overlap between partnership and capacity is real, but the panel wants relationship depth weighted heavily in this round. That is a decision, recorded as a decision.

The failure mode here is volume. A model asked to critique will always find something. Twenty concerns on a five-row rubric buries the two that matter, and the panel stops reading by the third page.

Cap the output. Ask for the three routes a candidate is most likely to take, ranked by how easy they are and how much score they move. A route that requires forging an audited document is not the same risk as one that requires generous rounding.

The second failure is a critique that ignores the field. Models default to a general theory of measurement and miss the specific thing everyone in the sector already knows is gamed. Feed in the last round's applications, or the notes from the previous panel, and the critique gets concrete. Historical decisions and their outcomes are the best fuel for this.

The third is scope drift into strategy. A critique that questions whether the fund should exist is not a criteria critique. Bound the request to the rubric in front of it. Strategy arguments belong upstream, in the pre-mortem on the whole round.

Nothing changes in the rubric until a named person accepts the change. The critique is advice from a system with no stake in the round. The chair has the stake, the history with the board, and the obligation to defend the scores in six months.

Record what was rejected and why. A concern the panel considered and set aside is different from one nobody saw. When a candidate later exploits exactly the route the critique named, the file should show that the panel weighed it.

Freeze the rubric once the round opens. Mid-round edits make early and late candidates incomparable, and the fix is worse than the flaw. If the stress test arrives late, note the weakness, run the round, and change the rubric for the next one.

The last check belongs to whoever will score. Give two reviewers the revised criterion and a real candidate. If they read the row differently, the wording is still broken, whatever the critique said. Reviewer disagreement on the same text is the only test that matters, and it is cheaper than rescoring the candidates you dropped.