Criteria-based output scoring

Criteria-based output scoring turns a finished work product into a structured QA report. An LLM reads the completed task output and a decomposed acceptance checklist, then scores each criterion with a clear pass, fail, or insufficient-evidence judgment and a short rationale.

This is not a free-form quality opinion. The checklist defines what “done” means for that task type. The model’s job is to apply those criteria consistently, cite what it saw (or did not see) in the output, and surface gaps before a human QA lead signs off.

Related checks that often sit next to this pattern: Adversarial review pass, Automated completeness check, and Policy compliance scan. Completeness asks whether required pieces exist. Policy scan asks whether constraints were respected. Adversarial review probes for weaknesses. Criteria-based scoring asks how well the output meets the acceptance bar you already defined.

Use this pattern when work arrives in batches and each item must clear the same acceptance bar: tickets closed against an SOP, tickets marked ready for handoff, content or reports marked complete, or operational packets assembled for the next team.

It fits best when:

• Acceptance criteria can be written as discrete, testable checklist items rather than a vague “looks good.”
• Outputs are text-rich enough that a model can inspect them (documents, tickets, structured fields, attached notes).
• Volume makes line-by-line human scoring slow, but you still need a defensible trail for each item.
• QA owns final disposition: approve, return with defects, or escalate.

Skip or narrow the pattern when criteria are still unstable, when the “output” is mostly binary system state with no inspectable artifact, or when scoring would require ground truth the model cannot see (physical inspection, live system access, or confidential side channels). In those cases, keep the checklist for humans and automate only the parts that are observable.

The scorer needs two primary inputs:

1. The completed task output (the artifact under review).
2. The acceptance checklist for that task type, decomposed into individually scoreable criteria.

If either input is missing, empty, or unresolved, return empty output. Do not invent a checklist from memory. Do not score against a partial or guessed set of criteria. Do not produce a “soft fail” narrative that implies a review happened when the prerequisites were not present. Empty output is the correct signal that the pipeline cannot run yet.

Before scoring, also confirm:

• Checklist version and task-type binding (so yesterday’s SOP is not applied to today’s process).
• Output identity (task ID, revision, or hash) so the report attaches to the right artifact.
• Scoring scale and required fields for the report schema your QA tooling expects.

A practical workflow for an operations QA lead looks like this:

1. Normalize the checklist. Each criterion should be atomic, observable, and worded so pass/fail is meaningful. Prefer “Includes a root-cause statement with supporting evidence” over “Root cause is good.”
2. Bind the output. Load the completed artifact and any allowed supporting fields (status, timestamps, linked references) that the checklist explicitly permits the scorer to use.
3. Score criterion by criterion. For each item, the model returns a judgment, a confidence or evidence-strength note if your schema uses one, and a short rationale grounded in the output text or structured fields.
4. Aggregate without hiding detail. Produce an overall status (for example: ready for QA sign-off, needs rework, blocked by missing evidence) while keeping per-criterion rows intact. Aggregation should never replace the checklist trail.
5. Route to human sign-off. The QA lead reviews the structured report, especially fails and insufficient-evidence rows, then approves, rejects, or requests a targeted fix.

Human-in-the-loop is mandatory. The model scores against the checklist; QA still signs off. Treat model judgments as decision support, not release authority. Calibration sessions help: sample scored items, compare against senior QA judgment, and tighten criterion wording where the model and humans diverge for the same reason.

When criteria conflict or overlap, resolve that in the checklist design, not at scoring time. Ambiguous criteria create noisy reports and teach teams to ignore the score.

A useful report is boring in the best way: same shape every time, easy to scan, easy to audit.

Include at least:

• Task and checklist identifiers, plus checklist version.
• Per-criterion rows: criterion ID or label, judgment, rationale, and optional excerpt or field pointer.
• Summary counts: passed, failed, insufficient evidence, not applicable (if your process allows N/A with an explicit rule).
• Overall disposition recommendation for the QA queue.
• Explicit “not evaluated” handling only when a criterion is marked out of scope by the checklist rules, never because the model skipped it.

Rationale quality matters more than numeric polish. A fail without a pointer to what was missing forces the QA lead to re-read the whole artifact. A pass that merely restates the criterion adds noise. Prefer short, concrete notes: what was present, what was absent, and which checklist language drove the call.

Insufficient evidence is a first-class outcome. If the output does not contain enough material to judge a criterion, say so. Collapsing that into a fail can send work back for the wrong reason; collapsing it into a pass hides risk.

Common failure modes:

• Checklist drift. The live process changed, but the scorer still uses an old list. Bind versions and fail closed (empty or blocked) when versions do not match.
• Over-scoring from tone. Fluent writing can look complete while missing required operational facts. Keep criteria fact-oriented and require explicit evidence pointers.
• Hidden double standards. Different teams encode different bars under the same task name. Separate checklists by task type and owner.
• Treating scores as automatic release. That breaks accountability. Keep sign-off with QA.
• Scoring without inputs. Missing output or missing checklist must yield empty output, not a fabricated review.

Operational practices that keep the pattern trustworthy:

• Version checklists like code; note what changed and why.
• Spot-check a fixed sample of scored items each week, with special attention to borderline cases.
• Feed recurring fail patterns back into training, templates, or upstream completeness checks so the same defects are not scored forever.
• Keep policy and adversarial passes distinct when the question is “allowed?” or “what could go wrong?” rather than “does this meet acceptance?”

When the acceptance bar is clear, the artifact is present, and a human still owns the stamp, criteria-based output scoring gives operations QA a consistent, auditable first pass over completed work without pretending the model replaced judgment.