Theory-of-change generator + validator

A theory of change is a set of arrows. Inputs make activities possible, activities produce outputs, outputs are meant to cause outcomes. The interesting part is never the boxes. It is whether each arrow survives contact with evidence.

So the useful output has two halves. A drafted chain, written from the candidate brief in the program's own terms. And a verdict per arrow: supported by a named source, contested, or unevidenced. An unevidenced arrow is not an error. It is the assumption the program is asking the funder to accept.

Assumptions get their own list, stated as sentences somebody could disagree with. "Participants can attend evening sessions" is an assumption. "Strong community engagement" is a mood. The validator should refuse the second form and ask for the first.

This sits at shallow assessment because it is cheap and it is a filter. A chain with four unevidenced arrows in a row does not need a deep dive to look risky. It needs a conversation with the applicant.

Do not generate a theory of change from a mission statement. Load the candidate brief as text, plus whatever the applicant submitted as a logic model. If the brief never names a population, a mechanism, and a time horizon, stop. The generator will otherwise produce a competent chain for a program nobody described.

Three fields decide most of the quality. The target population, defined tightly enough to count. The mechanism, meaning the specific thing the program believes causes the change. And the outcome window. A chain to a twelve-month wage effect is not the same as one to a five-year effect.

Set the evidence corpus at the same time. Name which literature, internal evaluations, and applicant documents the validator may cite. A validator with open web access will find something for every arrow. That is worse than finding nothing, because it looks like verification.

Criteria written earlier in the round should carry through. If the screening bar came out of a brief converted into a hypothesis tree and criteria, the outcome layer should use the same wording. Two vocabularies for the same outcome make the chain impossible to compare across candidates.

Generate the layers first, in order, from the brief alone. Inputs, activities, outputs, outcomes, assumptions. Keep each box short and concrete enough to argue with. A box that reads "capacity building" should be rejected before validation starts.

Then break the chain into individual arrows and test them one at a time. Batch validation of a whole chain returns a paragraph of praise. Arrow-level validation returns a table with holes in it. The holes are the deliverable.

Each arrow gets the same treatment. State the causal claim as a sentence. Search the named corpus for evidence about that specific claim, in a comparable population. Report what was found, where, and how close the setting is to this one. Then mark the arrow supported, contested, or bare.

Population mismatch is the thing to record most carefully. Evidence about mentoring and retention among university students is real evidence. It is not evidence about adults in a trades program. Note the mismatch on the arrow rather than discarding the source.

Contested is a distinct verdict and it should stay distinct. It means the literature disagrees, which is different from silence. Collapsing both into "weak evidence" hides which one a reviewer can resolve by reading.

This scenario is illustrative, with invented details. A regional workforce fund is assessing a candidate apprenticeship program in the building trades. The brief promises higher wages for participants two years after completion. Tomas Berg, the fund's evaluation lead, runs the generator on the brief.

The drafted chain has four activities and six arrows. Inputs are a monthly stipend, employer placements, and a tools grant. Activities are recruitment, twelve weeks of classroom instruction, placement, and a mentor pairing. Outputs are enrollments, completions, and placements. The outcome is a wage gain at twenty-four months.

Validation returns different verdicts on each arrow. Stipend to completion is supported, with three evaluations of stipend programs in comparable populations. Classroom instruction to competency is supported but narrow, resting on one study of a different trade. Placement to sustained employment is contested. Two cited evaluations disagree on whether placements persist past the first year.

Two arrows come back bare. Mentor pairing to completion has nothing in the named corpus about this population. Sustained employment to a wage gain at twenty-four months also has nothing. The applicant's own pilot stopped measuring at twelve months.

Five assumptions are extracted, and one of them is the whole program. It reads: employers will keep apprentices after the subsidized placement ends. Nobody had written that sentence down before the validator produced it. That single line reframes what the fund needs to ask about.

The dominant failure is a validator that wants the chain to work. Asked whether evidence supports an arrow, a model will find the most supportive reading available. Prompt for disconfirming evidence explicitly, as a separate pass, with its own output column. One pass for support and one for contradiction produces an honest table.

The second failure is a chain that reads as if the program wrote it. Generators trained on grant language reproduce grant language. Boxes come back as "strengthened pathways" and "increased readiness". Neither can be validated, because neither says what would happen to whom.

Skipped layers are the third. Applicants often jump from outputs straight to a long-term outcome. A validator that accepts that jump has skipped the arrow that carries all the risk. Force an intermediate outcome, or mark the gap.

Evidence that does not say what the citation claims is the fourth. A source can be real, recent, and about the wrong intervention. Run the cited passages through a check that each citation supports its claim before the table travels.

The last one is criteria that cannot fail. If the outcome layer is defined so loosely that any result confirms it, validation is theater. A critique of proposed criteria for gameability and overlap catches that before the chain is drawn.

A bare arrow is not a rejection. Every worthwhile program rests on something nobody has tested yet. The evaluation lead decides which untested arrows the fund is willing to carry. That decision goes in writing, with a name on it.

The validator has no view on risk appetite. It reports what the corpus said and where it was silent. A model that ranks candidates by count of supported arrows is answering a question nobody asked. Six weak arrows in a familiar field can be safer than one weak arrow in a new one.

Send the bare arrows onward rather than closing them. Each one is a research question with a known consumer, which is the shape an unresolved uncertainty packaged as a brief needs. Arrows that matter enough to test now go to deep assessment instead. There, a heterogeneous adversarial review of the draft will attack the same chain harder.

Keep the chain attached to the candidate record. It belongs beside the compiled one-pager for that candidate, not in a separate folder. A year later, the useful question is which arrow the fund bet on. That answer only exists if somebody recorded it at the time.