SOP gap identification

SOP gap identification compares how work was actually done against how the procedure says it should be done. An LLM reads a structured execution log (or an equivalent step-by-step record) and the current standard operating procedure, then lists steps that were skipped, reordered in a way that breaks dependencies, or replaced with improvised actions that are not in the SOP.

The output is a gap report, not a revised procedure. Each flagged item names the SOP step, what the log shows instead, and why the difference matters for quality, safety, or handoff. Process owners decide whether the SOP is wrong, training failed, tools blocked compliance, or the improvisation was justified for that case.

Related work often sits nearby: lessons-learned extraction captures what the team discovered after the fact, knowledge base update trigger routes confirmed changes into controlled content, and procedure retrieval at task start reduces drift by putting the right SOP in front of the operator before work begins.

Use this pattern in the operations close stage, when a job, shift, or incident cycle has produced an auditable trail and you need to check quality of adherence before you close the loop. Typical owners are process owners, quality leads, and ops managers who maintain SOPs and review exceptions.

Good fits include:

• Post-job reviews where technicians or coordinators logged steps in a ticket, work order, or checklist system
• After-action reviews where the team reconstructed the sequence from chat, notes, and system events
• Recurring procedures with known improvisation (workarounds for broken tools, tribal shortcuts, “we always skip that in practice”)
• Quality holds where a defect or customer complaint raises the question of whether the SOP was followed

Poor fits include live coaching during the task (that is closer to procedure retrieval and guided execution), pure document comparison without an execution record, and cases where the only evidence is memory with no log.

Two inputs are required. If either is missing or unusable, return empty output and state which input failed. Do not invent steps, do not fill gaps from general knowledge, and do not “guess” the SOP from similar procedures.

Execution record. Prefer a step-ordered log with timestamps or sequence numbers, actor, and action text (or structured checklist completions). Acceptable substitutes include work-order step histories, ticket timelines with explicit step markers, or a reconstructed sequence the process owner has already validated as the record of what happened. Free-form narrative alone is weak input; if used, the model should only flag gaps that are clearly stated as actions taken or omitted, and should mark low-confidence items for human review.

Stated SOP. Use the authoritative version that applied at the time of execution (version ID and effective date when available). The SOP must expose ordered steps or mandatory checkpoints. Glossaries, training decks, and informal wiki notes are not substitutes for the controlling SOP.

Empty output when:

• No execution log (or equivalent validated sequence) is available
• No SOP (or wrong version with no way to resolve which procedure applied) is available
• The SOP has no discrete steps to compare against
• The log cannot be aligned to any procedure (wrong job type, wrong site playbook, or ambiguous procedure ID)

When output is empty, say so in one sentence naming the missing input. That keeps reviewers from treating silence as “no gaps found.”

Alignment comes first. Map log events to SOP steps using step IDs, shared verbs and objects, and sequence. Ambiguous mappings stay out of the hard-fail list and go into a review queue.

Then classify differences:

1. Skipped mandatory step. The SOP requires the step; the log shows no completion and no documented waiver.
2. Improvised step. The log shows an action that is not in the SOP (or replaces a named step with a different method).
3. Order violation. Steps ran out of sequence in a way that breaks a stated dependency (for example, sign-off before inspection).
4. Partial completion. A step appears started or partially noted but lacks the required evidence or sub-checks the SOP demands.

Each flag should be self-contained: SOP step reference, evidence from the log (quote or structured field), classification, and a short rationale. Prefer precision over volume. One clear skip beats ten speculative mismatches.

Human-in-the-loop is non-negotiable. The model flags skipped or improvised steps; process owners still change the SOP, open training actions, or accept a documented exception. Auto-editing procedures from model output creates uncontrolled document drift and hides ownership.

A practical report for a process owner usually contains:

• Job or case ID, SOP ID and version, and time window of the execution
• Summary counts by classification (skipped, improvised, order, partial), without implying a score or grade unless your quality system already defines one
• A table or list of findings sorted by risk or by SOP step order
• Explicit “not found / empty” when inputs were insufficient
• Open questions for the owner (for example, “Was a verbal waiver recorded elsewhere?”)

Keep tone factual. Avoid blame language. The goal is to surface procedure drift so you can fix the system: outdated SOP, missing tooling, unclear step wording, or training gaps.

Downstream, confirmed gaps feed two paths. If practice was correct and the SOP is stale, route a controlled update via your knowledge base update trigger workflow. If the SOP was right and practice drifted, capture durable lessons with lessons-learned extraction and reinforce the correct path with procedure retrieval at task start on the next cycle.

This use case fails when the log is incomplete by design (people only log exceptions), when multiple SOPs apply and the wrong one is attached, or when improvisation was required for safety and never written down. Incomplete logs produce false “skips”; wrong SOP versions produce false improvisation flags.

LLMs can over-match synonyms (“verify” vs “check”) or under-match when operators use local jargon. Mitigate with a short glossary for the site or line, require step IDs in both systems when possible, and keep low-confidence items out of automated escalation.

Do not treat a clean gap report as proof of quality. Absence of flagged skips only means the available log aligned with the available SOP. Missing evidence still needs human judgment.