Outcome Evidence Extraction

Outcome evidence extraction turns unstructured case documentation into candidate indicator evidence that an M&E officer can review. Case notes, session records, and progress narratives often contain the signals funders and boards ask for: skill gains, service completion, protective factors, attendance patterns, or referral outcomes. Those signals sit in free text. Staff write for clinical or program continuity, not for indicator matrices.

An LLM reads each record against a defined indicator dictionary and surfaces candidate evidence: which indicator may apply, the supporting excerpt, and a short rationale. The model does not finalize results, close cases, or submit reports. M&E validates every candidate before it enters official reporting.

This page is for nonprofit M&E officers and data leads who already manage structured client or participant data in platforms such as Vera Solutions, and who need a repeatable way to pull measurable outcomes from narrative fields without reading every note by hand.

Extraction pays off when caseloads are large, notes are rich, and indicators are already defined. Typical conditions:

• Program staff record sessions, home visits, or coaching contacts in a CRM or case system with narrative fields.
• The measurement plan maps outcomes to indicators with clear definitions, units, and evidence criteria.
• Reporting cycles require evidence of change, not only activity counts.
• Manual review of notes for indicator evidence is the bottleneck, not the lack of a plan.

It is less useful when notes are sparse one-liners, when indicators are still being designed, or when evidence lives only in paper forms that never enter the system. In those situations, fix documentation quality or the measurement plan first.

Three inputs must be present together. If any one is missing, the run should return empty output rather than invent evidence.

Case notes and session records. The working set is the caseload slice under review: open cases, a cohort, a reporting period, or a site. Fields that usually matter include session date, staff author, service type, and free-text narrative. Attachments and scanned PDFs are out of scope unless they are already transcribed into text fields the pipeline can read.

Indicator dictionary. Each indicator needs an ID, plain-language definition, evidence criteria (what counts and what does not), preferred unit or status values, and any exclusion rules. Ambiguous definitions produce noisy candidates. Align dictionary language with how staff actually write about the work.

Caseload context. Stable identifiers for participants or households, program enrollment windows, and the reporting period bound the search. Without period and population bounds, the model cannot tell which notes are in scope.

Empty-output rule: if case notes, session records, or the indicator dictionary are missing, incomplete, or unreadable for the selected scope, return no candidates. Do not backfill from training data, prior cohorts, or similar programs.

Treat extraction as a batch assist over a defined scope, not as live coaching of caseworkers.

1. Select scope. Choose period, site or program, and caseload filter in the case system (for example Vera Solutions views or exports that include narrative fields and IDs).
2. Attach the dictionary. Load the current indicator definitions for that program. Version the dictionary so reviews can be reproduced.
3. Run extraction. For each note or session record in scope, the model proposes zero or more candidates: indicator ID, excerpt span, confidence label or score if you use one, and a one-line rationale tied to the dictionary criteria.
4. Dedupe and group. Collapse near-duplicate excerpts for the same participant and indicator. Prefer the clearest span over volume of hits.
5. Queue for M&E review. Present candidates in a review list filtered by indicator, site, or low-confidence first. M&E accepts, rejects, or edits before anything is marked reportable.

The model’s job stops at candidate evidence. Acceptance, aggregation into outcome rates, and narrative for funders remain human work.

Human-in-the-loop is not optional. Reporting, donor compliance, and participant dignity require that staff judgment sit on top of machine suggestions.

M&E (or a designated data quality lead) should:

• Confirm that accepted excerpts actually meet the indicator’s evidence criteria, including timing relative to baseline and enrollment.
• Reject candidates that confuse activity with outcome (for example, “attended workshop” offered as proof of skill mastery).
• Spot systematic bias: some workers write detailed notes; others do not. Extraction will over-represent talkative documentation unless you monitor coverage by worker and site.
• Decide how accepted evidence maps into the official store: status fields, outcome events, or linked attachments, per your data model in Vera Solutions or equivalent.
• Keep an audit trail: who accepted what, against which dictionary version, for which reporting period.

Program managers may use rejected patterns to coach documentation, but coaching is separate from locking results for a grant report.

Missing or thin notes. Empty or near-empty narratives should yield empty candidates for that record, not inferred outcomes from demographics or enrollment alone.

Dictionary drift. If indicators change mid-year and the run uses an old dictionary, candidates will not match what leadership reports. Pin dictionary version to each batch.

Over-extraction. Models tend to find weak matches. Prefer precision settings and require M&E acceptance; do not auto-write outcome fields.

Privacy and minimization. Limit the text sent to the model to fields required for evidence review. Mask or exclude sensitive free text that is irrelevant to the indicator set when policy requires it.

Cross-caseload leakage. Process one scoped batch at a time. Do not let prompts or context windows mix participants in a way that attributes Person A’s note to Person B.

When in doubt, fail closed: no notes or no dictionary means no output.