Skip to main content
DoneThat

AI Adoption GuideGovernmentClose

Lessons learned extractor

LLM synthesizes inspection reports, incident logs, and contractor records into structured lessons learned for program evaluation and planning reuse.

Government processPlanFundAuthorizeDeliverInspectEnforceReportClose

By Don, DoneThat’s AI coach · updated

A cited lesson is the only acceptable output

Every extracted lesson must name the inspection report, incident log, or contractor record that supports it. If the model cannot point to a file and a passage, the sentence is not a finding. Discard it.

Program evaluation still owns the finding. The extractor rearranges what closeout files already say. It does not decide that a delay came from weather, staffing, or a weak specification. It does not invent a root cause because the template has a cause field. When the files are silent, the field stays empty.

Closeout packets are large and split across systems. Inspection narratives, daily logs, change orders, and contractor correspondence often sit in Microsoft document libraries, Palantir case workspaces, Accela permitting files, or Tyler records systems. Those platforms hold the artifacts. They do not make an uncited synthesis true.

A lesson with no source is the first failure mode. It reads like program language and looks finished. Nobody can check it. Send the evaluator back to the file instead of putting that sentence in the evaluation.

Assemble the closeout files before you extract

Load files, not a story you already believe. The useful minimum is the inspection reports that closed the work, the incident or occurrence logs for the period under review, and the contractor records of what was promised, changed, or disputed. Add punch lists, nonconformance reports, or hold correspondence if you have them. Do not ask the model to recall files you did not load.

Keep names, dates, and authors that a citation can follow. A generic final.pdf is harder to defend than a dated inspection file with a page or section the extractor can quote. Do not pre-summarize the packet into a briefing and then extract from the briefing. The model should see the same artifacts an evaluator would open.

Scope the load to one program, phase, or asset. Mixing two contracts produces general lessons that cite nothing specific. Keep residual issues and incomplete handover in their own packets and send them to the residual liability identifier or the asset handover documentation generator. Do not fold those streams into lessons learned.

You are not writing the inspection report. If a site visit still needs a structured write-up, use the inspection report auto-drafter. Extraction starts after those reports exist as files.

Extract with citations and leave silence empty

Instruct the extract so every lesson cites a source report or log, restates only what that source supports, and leaves root cause, recommended action, and similar fields blank when the files do not state them.

Ask for structured rows, not a memo. A workable row has the lesson statement, the condition or event, the source file and locator, a short excerpt, and a note about the text: stated, implied, or absent. Implied is a warning. Absent means blank. The note is not a score of whether the program succeeded.

One illustration. An inspection report notes that deck pours stopped twice because the temperature log fell below the specification floor. A contractor daily report records idle days on the same dates. A fair extract says pours paused when recorded temperatures fell below the specified minimum, citing the inspection section on pour holds and the daily reports for those dates. It does not add that the contractor failed to plan winter operations unless a loaded file says so. The readings and the idle days are in the record. The motive is not.

Leave blanks on purpose. If incident logs describe what happened and never name a cause, keep the cause empty. If contractor records dispute an inspection comment and the files you loaded never resolve the dispute, extract both statements with cites. Do not pick a winner. Do not merge them into one lesson.

Inventing a cause is the second failure mode. It shows up as helpful completion when a why column exists, so the model fills it. A filled why that is not in a report or log is fabrication. Delete it. A blank is better than a plausible story.

The evaluator accepts the finding, not the model

Acceptance is evaluation work. The lead, or the evaluator they assign, reads each row against the cited file, keeps what the source supports, edits wording to program terms, and rejects anything uncited, over-claimed, or causal without evidence.

Do not treat the extract as the evaluation. That is the third failure mode. A list of lessons is a working paper. It is not the program evaluation, not the leadership report, and not a judgment about significance, recurrence, or whether the next specification should change. Evaluation still decides that a cited fact can be true and still not worth carrying forward.

When you accept a row, keep the citation on the official finding. When you reject a row, record the reason: no source, source does not support the claim, cause invented, or duplicate of another cited lesson. That log is part of the evaluation trail. For a durable record of who loaded which files and who accepted which rows, use the audit trail auto-documenter instead of chat history.

Microsoft, Palantir, Accela, and Tyler can store the packet. They do not change the rule. The evaluator still opens the cited page. A system of record cannot own the finding.

Keep lessons extraction separate from other closeout products

Run the extractor at close, after inspections, incidents, and contractor records exist as citable files, and before you write the evaluation narrative. Build the narrative from accepted, sourced lessons, not from memory of a long project.

Keep the products distinct. Handover documentation describes what is turned over and in what condition. Residual liability work flags what remains open or disputed. Inspection drafting captures the site record. Lessons extraction only synthesizes what those records already support for evaluation and planning reuse.

If the point is planning reuse, an accepted lesson should be specific enough to change a checklist, a specification clause, or an inspection hold point. A lesson that says communication should improve, with no cite, cannot do that. A lesson that says pours were held when temperature logs fell below the specified minimum, per inspection report X and daily report Y, can, if evaluation decides the hold point needs a clearer winter protocol. The extractor supplies the cited material. Evaluation decides whether it becomes practice.

Stop when the files are silent. Do not run a second pass to fill gaps. A pass that invents root causes or writes lessons for issues never recorded fails the quality bar. Quality is a lesson that cites the source report or log, or an empty field that stays empty.

Is this worth automating for you?

Whether this pays back depends on how much time it takes your team today. Most teams estimate that from memory, and the estimate is usually wrong in one direction or the other.

DoneThat reconstructs where the time actually went, with no timers to forget, so you can measure the baseline before committing to a project and check the gain afterward.

Measure the baseline first