Hypothesis tree + criteria translator

Analysts are usually handed a paragraph. Someone senior wants to know whether a policy is working, or which of several options deserves money. The output that makes that paragraph workable is two documents. A tree that breaks the question into branches that do not overlap and together cover the question. And a screening sheet where each criterion has a threshold and a source.

The screening sheet is the harder half and the one teams skip. A criterion reading "strong evidence base" is a mood. A criterion reading "at least two evaluations with a control or comparison group, published or registered, on a population of this size or larger" can be applied by someone who has never met the commissioner.

Neither artifact is an answer. The tree says where to look. The screen says what will count. Both are proposals for a human to argue with, and the argument is short because the objects are concrete.

This step pays for itself downstream. Every hour spent making a branch exclusive is an hour not spent later untangling why two sub-questions returned the same sources. The dedupe pass on candidates cleans up records. It cannot clean up a tree whose branches were always the same branch.

The brief tells you what someone wants. It rarely tells you what the answer will be used for, and that is the input the tree needs. Ask three questions before generating. What decision follows this work. Who has to be persuaded. What is already fixed and cannot be reopened.

Constraints shape branches more than the topic does. If the budget envelope is set, a branch on whether to spend more is dead weight. If the legal basis for one option is unsettled, that branch needs a different kind of source and a different threshold. State the constraints in the prompt and the tree comes back narrower and more useful.

Also load the time and access budget. A criterion that requires a bespoke survey is not a screening criterion in a six-week study. Say how long the round runs and which data the team can actually reach. Otherwise the screen comes back elegant and unrunnable.

If the decision behind the brief is genuinely unknown, do not generate a tree. Go back and ask. A tree built on a guessed decision is worse than no tree, because it looks authoritative and the team will work inside it for weeks.

Build the tree in one pass and the criteria in a second. Doing both at once produces criteria attached to branches that will not survive review, and the wasted effort is in the criteria.

Test exclusivity with a single question per pair of branches: could one piece of evidence confirm both. If yes, the split is not real. Test coverage with the opposite question: what explanation of the observed problem has no home in this tree. A branch labeled other is an admission, not a solution, and it should force a redraw.

Then convert each leaf into criteria with four fields. The quantity or property being measured. The threshold or band. The source that will supply it. And the rule for a missing value. That fourth field is the one that gets left out, and it decides half the outcomes in a real screen.

Decide in advance whether a blank fails, passes, or parks the candidate. Most teams want park. A candidate missing a value is not a candidate that failed, and treating the two the same quietly filters out everything poorly documented. Hard-fail rules belong in the rules engine that knocks candidates out before scoring, where they are visible.

The scenario below is invented. A transport agency's analysis unit receives one paragraph from a director. Rural bus patronage has fallen for three years despite a subsidy that has not been cut. The director wants to know whether the subsidy should be restructured before the next funding cycle.

The analyst adds the constraints. The subsidy envelope is fixed for two cycles. Route-level service contracts run to 2029 and will not be reopened. The unit has six weeks and access to ticketing data, contract records, and the national travel survey.

The tree comes back with four first-level branches. Demand has moved, so the same service now carries fewer people. Supply has degraded, so the service on offer is worse than the contract implies. The subsidy is reaching the wrong routes. And the measurement changed, so the fall is partly an artifact of how patronage is counted.

The fourth branch is the one an analyst would have added on day nine, after a week of wondering why two data series disagreed. Getting it on the board in hour one is most of the value of this step. It is also the only branch that can be closed quickly, by comparing the counting method across the three years.

Criteria for the supply branch come back with thresholds. Scheduled departures delivered, measured from operator returns, with a band the unit already uses in contract monitoring. Mean wait at the twenty lowest-frequency stops, from timetable data, with a blank rule of park. Vehicle age, from the fleet register, with no threshold because nobody has agreed one. That last row goes to the director as an open question, not as a guessed number.

The characteristic failure is a tree that is symmetrical rather than exclusive. Models like four branches with three children each. Real questions are lopsided. One branch usually carries most of the risk and deserves four levels while another needs one. Symmetry is a sign nobody tested the splits.

The second failure is a threshold with no basis. A model will happily write "above 60 percent" because criteria look finished with a number in them. Ask where the number came from. If the answer is nowhere, replace it with a blank and a note, and let the team set it. Prior decisions and what they implied are a better place to find a real bar.

The third is criteria that restate the branch. A branch about whether the subsidy reaches the wrong routes becomes a criterion reading "subsidy targeting is appropriate." Nothing was translated. A real criterion names a quantity someone could pull tomorrow: subsidy per boarding by route decile, from contract records.

Run the draft tree through a critique of the criteria themselves before anyone starts collecting. The two steps catch different things. One checks the structure, the other checks whether the rows can be gamed.

The person who wrote the paragraph has to see the tree and say whether it is their question. This takes twenty minutes and it is the only reliable way to find out that the director meant something narrower. They will often add a constraint nobody wrote down.

The analyst owns the thresholds. A number in a screening sheet will be quoted in a meeting as though the unit chose it deliberately. Every threshold needs a person behind it, or a visible blank saying the team has not decided.

Version the tree and date it. Branches die during a study, and the record of which branch was closed by which evidence is the spine of the final write-up. A tree that quietly changes shape leaves the reader unable to tell what was tested and what was dropped for time.

Sourcing does not start until the tree is signed. Search is cheap and endless, and an unsigned tree turns into a corpus nobody can defend. Once the branches are agreed, each one becomes a query with an expected source class, and the round has a shape.