Forecasting tournament on outcomes

A forecasting tournament asks the people who decide to write down what they expect. Then it waits, and scores them against what happened. The scores are about the forecasters, not about the candidates. That distinction survives or the whole thing turns into a popularity vote with extra steps.

Nothing here ranks candidates. A candidate with a low predicted success probability may still be the right commitment. It might be cheap, reversible, or the only option in a category that matters. The tournament measures how well the committee reads situations, over years.

Effort is high because the value arrives late. A twenty-four-month question written today pays out two years from now. Somebody has to still be running the process then. Most tournaments die in month seven, when the person who set it up changes roles.

The minimum viable version is small. Two questions per decision, five forecasters, one resolution date each. A tournament with forty questions and no owner resolves none of them.

Write questions that a stranger could score without asking anyone what was meant. That means a named observable, a source of truth, and a date. "Will the program succeed" fails all three. "Will the utility report non-revenue water below the stated threshold in its twenty-four-month filing" passes.

Name the resolution source when the question is written, not when it comes due. A question resolved by whichever document is convenient in two years resolves to whatever the resolver wants. If the source might not exist, say what happens then. Ambiguous is a legitimate resolution state and it should be defined in advance.

Two horizons are usually enough. Twelve months catches execution. Did the work start, did the team form, did the first milestone land. Twenty-four months catches early outcome, which is where committees are usually wrong.

Prior decisions tell you which questions are worth asking. Reading what the committee previously approved and what happened, through surfacing the implicit bar in past calls, shows where its judgment already failed. Forecast those things, not the ones everyone gets right.

Let a model draft the questions and a person approve them. Drafting twenty candidate questions from a deep dive is fast and mechanical. Deciding which two are worth two years of attention is not.

Forecasts have to be locked before the decision is announced. Once the room knows the outcome, the forecast measures agreement with the chair. Register privately, close submissions, then hold the meeting. This ordering is the single most important operational rule here.

Collect probabilities, not verdicts. A number between zero and one carries information that yes or no discards. Let forecasters write a short rationale alongside, because the rationale is what you read later. A wrong forecast with a clear reason teaches more than a right one with none.

Score with a proper rule, chosen once and never changed mid-tournament. Brier scores are the usual choice and they are adequate. Compare each forecaster against the group median and against a naive base rate. Beating the base rate is the bar that matters; beating colleagues is a distraction.

Resolution needs a person and a calendar entry. Assign a resolver per question at registration, with a named backup. Automate the reminder, because nothing else will produce it two years out. An unresolved question is worse than no question, since it taught nobody anything.

Individual scores accumulate into something more useful than a leaderboard. Patterns of systematic optimism or pessimism per person are the actual product, and that is the job of tracking rater scores against outcomes.

This walkthrough is hypothetical, including every figure. A development finance institution is committing to a municipal water utility improvement program. Leo Nakamura runs the forecasting process for the investment team. Seven committee members forecast on each decision.

Three questions are registered before the committee meets. At twelve months, will the metering rollout reach the contracted district count? The utility's quarterly report resolves it. At twenty-four months, will reported non-revenue water fall below the agreed threshold in the audited filing? And at twenty-four months, will the tariff adjustment in the plan have been approved by the regulator?

The submitted probabilities cluster tightly on the first question and spread widely on the third. Six forecasters put the metering rollout above eighty percent. On the tariff question the range runs from fifteen to seventy. That spread is itself a finding, and it surfaces before the decision rather than after.

Two years later the results are uneven. The rollout landed roughly on schedule and most forecasters were close. Non-revenue water improved but missed the threshold, and the committee had been broadly too optimistic. The tariff was not approved, and the two forecasters who had been below thirty were right.

Their rationales both named the same thing: an election in the intervening period. Nobody else had mentioned it. That is the concrete output of a tournament, and it changes what the next appraisal asks about.

Ambiguity at resolution is the most common way these collapse. A question that seemed clear in year one meets a world that reorganized the reporting. The district count is now measured differently and nobody can say whether the target was met. Write the resolution criteria as if the person applying them will be hostile.

Participation decays quietly. Round one has seven forecasters, round six has two, and nobody announced the change. Report participation rate alongside every score. A calibration finding drawn from two people is a fact about two people.

Selection bias limits what the scores can mean. Committees only see outcomes for the candidates they funded. Whether the rejected ones would have done better is unobservable. Calibration measured only on approvals is calibration on half the distribution.

Scoreboards change behavior in the wrong direction. Forecasters who are ranked publicly start hedging toward the median. The median is exactly the thing you wanted them to challenge. Keep individual scores private and share the aggregate patterns.

Two harder problems have no clean answer. Forecasters sit on the committee, so their forecasts can influence the outcome they predicted. And two years is long enough that staff turnover erases the context behind old rationales. Both are reasons to keep the question count low and the records verbose.

A forecast is an input to a named decision-maker, and it stays that. The committee chair approves the commitment, signs the memo, and answers for it. No aggregate probability has authority over that signature. A rule that automatically blocks a deal below some median has moved the decision into a spreadsheet.

The strongest legitimate use is timing and structure, not approval. A wide spread on one question is a reason to stage the commitment. A confident low probability is a reason to ask what would change it. Both are conversations the committee has, with the forecasts on the table.

Calibration findings inform how a committee reads its own confidence. If the group has been consistently optimistic about regulatory approvals, that belongs in the room next time. It does not license anyone to adjust a colleague's number after the fact.

Forecasts pair naturally with how the decision itself was recorded. A committee already writing down decisions as probability distributions has most of the registration infrastructure in place. The tournament adds the resolution step and the scoring rule.

Resolution needs ground truth from the field, which somebody has to collect. Structured check-ins with the recipient on what held and what did not supply most of it. When resolution shows a core assumption was wrong, the underlying analysis is stale too. That is the moment to fire a refresh trigger on the claims that have gone out of date.