Deployment runbook generation

Release engineers spend too much time reconciling a generic checklist with what this environment actually runs. Product documentation describes intended behavior. The CMDB names the configuration items that will change. Past deployment logs show what broke last time and which verification steps caught it. An LLM can assemble those inputs into a step-by-step runbook for a specific change window without replacing your release tooling or approval flow.

The point is speed with traceability, not automation of execution. Each runbook step should carry a citation to a doc section, a CMDB configuration item, or a log event that justified the step. If the model cannot tie a step to one of those sources, the citation field stays empty. The release still executes. Operators proceed with human judgment on uncited steps rather than blocking the window because the draft is incomplete.

This pattern fits teams that already maintain docs in a wiki or repo, record CIs in ServiceNow or an equivalent CMDB, and retain Jenkins or GitLab pipeline logs from prior releases. Ansible playbooks and inventory exports can supply environment-specific commands when your docs reference them explicitly. The generator does not deploy, does not approve changes, and does not fabricate rollback commands when logs and runbooks offer no evidence for one.

Treat runbook generation as a retrieval problem first and a writing problem second. Gather three bundles and keep them labeled so citations stay auditable.

Product documentation. Export or paste the sections that cover the component being upgraded: prerequisites, configuration keys, health checks, known limitations, and post-deploy validation. Include version-specific release notes when the change spans a minor or major bump. Strip navigation chrome; the model needs paragraph text and headings, not site menus.

CMDB and change record context. Pull the configuration items attached to the change ticket: hostnames, load balancer pools, database instances, feature flags, and dependency links. If your process lives in ServiceNow, export the affected CI list and environment attributes (prod, staging, region) as structured text or CSV the model can reference by CI name or sys_id. The runbook should name the same CIs the change advisory board expects on the ticket.

Deployment logs from comparable releases. Select two or three prior pipelines for the same service and environment. Jenkins console output, GitLab job logs, and Ansible recap lines are useful when they show ordering, manual gates, and verification commands operators actually ran. Redact secrets and tokens before inclusion. Logs ground the runbook in practice; they are not a license to copy a rollback verbatim if none was performed.

Optional but valuable: run a pre-flight configuration validator on the target environment before generation so doc claims and live config mismatches surface as explicit warnings in the prompt, not as silent assumptions in the draft.

Prompt for a runbook schema your team already uses: step number, action, owner role, expected duration, verification, and a citation field. Instruct the model that every step must populate citation with exactly one of: doc:<section id or heading>, cmdb:<CI name>, or log:<pipeline id and line reference>. If no source supports the step, citation must be empty and the step must still appear if a human would reasonably perform it.

Illustrative example (payment API, prod, blue-green). Suppose docs state that /health must return 200 before traffic shifts, the CMDB lists lb-payments-prod-01 and api-payments-prod-b, and a March GitLab deploy log shows operators ran curl against the green pool before cutover. A generated step might read: "Verify green pool health via HTTPS GET /health on each instance behind api-payments-prod-b." Citation: doc:Post-deploy validation plus cmdb:api-payments-prod-b. A later step: "Shift traffic on lb-payments-prod-01 per load balancer runbook." Citation: cmdb:lb-payments-prod-01. If the model suggests "rollback by restoring previous container tag" but no doc section and no log entry describe that command, citation stays empty and the step text should not invent a tag name or registry path.

Review the draft in the same session as the change owner. Merge duplicate verification steps. Align CI names with the ticket. For communication-heavy windows, pair the runbook with an upgrade communication drafter so customer-facing text matches operational steps without conflating the two artifacts.

Generation ends where execution begins. Check the runbook into the change record or paste it into the war room doc. Operators mark steps complete in your existing tooling. Pipeline stages in Jenkins or GitLab still gate promotions; Ansible still applies what your playbooks define. Nothing in this workflow auto-deploys because an LLM produced a numbered list.

During execution, empty citations are first-class signals. A step with no cite is not forbidden; it is unverified. The release lead decides whether to perform it, skip it, or pause for a source. That decision belongs in the thread, not in a retroactive citation invented under pressure.

When scope shifts mid-window, regenerate from updated inputs rather than editing steps by hand without new citations. A change impact simulator can refresh which CIs and dependencies matter if the blast radius grows after a failed canary.

Post-release, attach the executed runbook (with checkmarks and timestamps) to the ticket. Note which steps ran without citations so the next generation pass can target those gaps in docs or logging.

Treating the draft as executed. A polished runbook in the change ticket is not evidence the work happened. Auditors and incident reviewers need pipeline IDs, approval records, and operator notes. If the only artifact is the generated markdown, you have documentation theater. Keep execution truth in systems of record; keep the runbook as the intended script.

Filling empty citations to "finish" the doc. Under time pressure, someone pastes a wiki link that does not mention the step or cites the wrong CI because the name looked close enough. That erases the purpose of mandatory citations. Empty fields are honest; false citations are hazardous. Fix the source material or accept explicit human sign-off on uncited work.

Inventing a rollback command. Models pattern-match on "rollback" and produce plausible shell that never ran in your environment. Unless product docs specify a rollback procedure and logs show it used successfully, the runbook should list decision criteria and point to a rollback decision assistant for structured triage, not a fabricated one-liner. Rollback without evidence belongs in a deliberate decision record, not in autogenerated imperative text.

Overfitting to one old log. A single anomalous Jenkins build that skipped tests should not become the canonical procedure. Prefer steps that appear across multiple logs or that docs require regardless of past shortcuts.

Stale CMDB. Steps tied to decommissioned CIs read confidently and fail loudly at 2 a.m. Reconcile CMDB exports at generation time; if a CI is missing from the export, cite that gap rather than guessing replacements.

Speed compounds when inputs improve. After each window, catalog citation gaps: steps operators performed with empty cites, doc sections that were wrong, CIs that renamed, logs that lacked verification detail. Feed that list to doc owners and pipeline maintainers.

Standardize log retention and labeling so the next runbook generation can reference log:gitlab:12345:deploy-prod:health-check instead of pasted excerpts. Align CMDB naming with infrastructure-as-code labels Ansible and your cloud console already use.

Runbook generation is a drafting accelerator for release engineers and SRE leads who already own change discipline. It does not remove approvals, does not click deploy, and does not replace rollback judgment. It gets a credible, environment-specific checklist in front of the team faster, with every step either anchored to evidence or visibly unanchored so humans know where to look.