Dataset Quality Monitoring

Outcome datasets often look complete until someone tries to build a dashboard, write a donor report, or reconcile figures across sites. A missing endline ID, a duplicated household row, an indicator coded outside its allowed range, or a baseline value that contradicts the measurement plan can all pass a quick visual check and still break analysis later.

Dataset quality monitoring uses an LLM to scan structured outcome data against an expected schema and indicator dictionary before those rows enter reporting tools such as Sopact Sense or similar impact platforms. The model surfaces candidate issues: inconsistent values, likely duplicates, missing required fields, out-of-range scores, mismatched categorical labels, and obvious unit or period conflicts. It does not rewrite the dataset or publish corrected numbers on its own.

The audience for this check is the M&E analyst who owns cleaning and sign-off. Program staff may collect the data. Leadership may consume the charts. The analyst still decides which flags are true errors, which are acceptable edge cases, and which need follow-up with field teams.

Run the check after collection or import, and before aggregation, dashboard refresh, or narrative reporting. Typical triggers include a new survey export, a merged multi-site file, a quarterly outcome upload, or a schema change in the indicator dictionary.

The review is most useful when three artifacts are available together:

1. The outcome dataset (tabular rows keyed to participants, households, sites, or cohorts).
2. A schema that defines required columns, types, and join keys.
3. An indicator dictionary that defines allowed values, scoring rules, units, reporting periods, and which fields are mandatory for each indicator.

Without those inputs, automated flagging has nothing reliable to compare against. The intended behavior is empty output: no issue list, no implied clean bill of health, and no invented schema. An empty result in that case means “inputs incomplete,” not “data looks fine.”

Pair this step with upstream and adjacent work. Measurement Plan Validity Review helps confirm that indicators and methods are coherent before large volumes of data arrive. Outcome Evidence Extraction and Qualitative Response Auto-Coding may feed structured fields into the same tables this check later inspects. Quality monitoring sits at the gate between those structured outputs and formal reporting.

Treat the model as a structured reviewer, not a silent cleaner. Provide the dataset sample or full extract, the schema, and the indicator dictionary in one review pass (or in chunked passes for large files, with the same rules applied consistently). Ask for a machine-readable issue list: row or key identifiers, field names, issue type, brief rationale, and severity or confidence notes. Keep the original file unchanged until a human applies fixes.

Useful issue classes for nonprofit outcome data include:

• Missing values: required baseline or endline fields blank; IDs present in one wave but absent in another without an exit reason.
• Duplicates: same participant or household ID repeated with conflicting outcomes; near-duplicate rows that differ only by whitespace or date format.
• Inconsistencies: categorical labels that do not match the dictionary (yes vs Y vs completed); gender or age fields that conflict across waves; site codes that do not appear in the site list.
• Range and logic checks: Likert scores outside the defined scale; percentages above 100; endline dates before baseline dates; outcome values that contradict skip logic implied by the schema.
• Period and unit mismatches: currency or count fields mixed without a unit column; reporting period tags that do not align with the measurement calendar.

The model should cite the rule it used when it can (dictionary allowed values, required flag, type constraint). When it cannot map a field to the dictionary, it should say so rather than invent a rule.

Human-in-the-loop is mandatory. Every flag is a suggestion for the M&E analyst. Corrections, suppressions, and “accepted as-is” notes stay with the analyst. The model may propose a likely fix (for example, normalize F/Female to the dictionary label), but applying that fix remains a human action, ideally with an audit trail in the same workflow used for Sopact or warehouse updates.

Flag volume is not the same as data quality. High volume can mean a bad export, a dictionary that drifted from the form, or a model that over-flags free-text fields it should ignore. Low volume can mean a narrow sample, not a clean population.

Analysts typically:

• Confirm whether a flagged duplicate is a true double entry or a legitimate multi-enrollment rule.
• Decide whether missing endline data is attrition that should be coded, not imputed.
• Update the indicator dictionary when the form changed and the dictionary lagged.
• Send targeted queries to partners or enumerators instead of guessing values.
• Approve a cleaned extract for reporting only after reviewing severity and coverage of the scan.

Document decisions next to the issue list: fixed, deferred, false positive, or needs field verification. That record matters for donor audits and for the next monitoring cycle, when the same false positives should be filtered or the dictionary updated so the model stops raising them.

Required inputs. Outcome dataset; schema (columns, types, keys, requiredness); indicator dictionary (definitions, allowed values, units, periods). Optional but helpful: site list, wave calendar, known attrition codes, and prior cleaning notes.

Empty output conditions. If any required input is missing, malformed beyond use, or clearly mismatched (dictionary indicators that do not appear in the schema, or a file with no identifiable keys), return an empty issue list and an explicit incompleteness reason. Do not fabricate columns, do not infer a full dictionary from column names alone, and do not mark the dataset as validated.

Limits. LLMs can miss rare logic errors, struggle with very large files unless chunked carefully, and over-interpret free text. They are weak substitutes for statistical outlier methods when you need distribution-aware detection. They also cannot replace field verification when the “correct” value is only knowable on site. Use deterministic checks (null rates, unique-key constraints, join integrity) alongside the LLM pass when those rules are already crisp in code.

Operational tip. Keep the dictionary versioned with the dataset version you are about to publish. A quality scan against yesterday’s dictionary on today’s form redesign will generate noise and erode trust in the flags.

• Measurement Plan Validity Review: stress-test indicators, methods, and reporting claims before collection scales.
• Outcome Evidence Extraction: pull structured outcome signals from reports and notes that later land in the same tables.
• Qualitative Response Auto-Coding: code open responses into categories that still need dictionary-aligned quality checks before reporting.