Rollback decision assistant

A rollback decision assistant watches deployment metrics in real time and surfaces a recommendation only when configured anomaly thresholds are crossed. The output is a structured judgment, not an executed rollback. On-call still owns the decision, the timing, and the communication with stakeholders.

Use this pattern when a release has moved past pre-deploy checks but post-deploy signals are ambiguous enough that experienced engineers would pause and compare numbers across a short observation window. It fits services where rollback is technically feasible, where metric baselines exist from before the change, and where the cost of a false rollback is lower than letting a bad deploy run unchecked for another ten minutes.

Skip it when telemetry for the affected service is incomplete, when the deploy path cannot be reversed cleanly, or when the team has not agreed on which metrics matter for rollback. In those cases the assistant should stay quiet rather than guess.

The assistant needs a current slice of deployment telemetry and a pre-deploy baseline for the same signals. Wire it to the observability stack your team already uses: Datadog for time-series and dashboards, Splunk for log-derived or aggregated operational metrics, or equivalent exports from GitLab deployment events paired with service-level measurements. ServiceNow can hold the change record, affected configuration items, and the link between a release identifier and the services under watch.

Define the observation window explicitly in configuration. A five-minute window after traffic shift is different from a thirty-minute soak period. The model compares values inside that window against thresholds your team set, not generic industry defaults.

Document which metrics are in scope: error rate, latency percentiles, saturation, queue depth, or custom business counters. If a metric is not wired, it does not appear in output. Empty fields stay empty. Do not backfill missing telemetry with inferred values or placeholder percentages.

Pair this wiring with outputs from post-deploy anomaly clustering when several signals move together. Clustering helps the assistant prioritize which correlated shifts belong in one recommendation instead of flooding on-call with isolated alerts.

Every recommendation that clears the anomaly gate must cite three elements: the metric name, the threshold that was crossed, and the observation window where the breach was detected. Optional narrative explains why those three facts matter together. Nothing else substitutes for them.

Metric name. Use the canonical name from your monitoring system so on-call can jump straight to the chart. "HTTP 5xx rate" beats "errors look bad."

Threshold. State the configured limit and the observed value or trend that violated it. "Threshold: 2% 5xx over 5 min; observed: sustained above 3.1% for 6 min" is actionable. "SLO breach at 47%" is not, unless your organization actually defines and exports that SLO with the same name in the same system. Never invent an SLO breach percent to make the recommendation sound authoritative.

Observation window. Give the start and end of the evaluation period relative to deploy completion or traffic shift. On-call needs to know whether the signal is still active or was a brief spike.

When telemetry is missing for a required metric, leave the corresponding field blank and note the gap in plain language. A partial recommendation with honest blanks is more trustworthy than a complete-looking sentence built on assumptions.

Illustrative shape (no live numbers)

Imagine a canary deploy for a checkout API. The assistant evaluates checkout.http.5xx_rate against a 2% ceiling over a ten-minute window starting at canary promotion. Error rate climbs above threshold at minute four and stays elevated. The recommendation reads:

• Metric: checkout.http.5xx_rate
• Threshold: 2% over rolling 10 min
• Window: T+4 min to T+10 min post-promotion
• Summary: Elevated server errors correlated with new build; no auto-rollback performed.

If latency metrics were not exported for that service, the latency line stays empty. On-call opens Datadog, confirms the chart, checks recent commits, and decides whether to roll back or hold while investigating a downstream dependency.

Treat the assistant as a ranked briefing, not a trigger. The recommended sequence:

1. Receive the recommendation in the same channel or ticket stream on-call already monitors alongside deploy notifications from GitLab or your release orchestrator.
2. Verify each cited metric in the primary observability tool. Cross-check logs in Splunk if the recommendation references log-derived counters.
3. Contextualize using the ServiceNow change record: scope of blast radius, prior rollback history, maintenance windows, and customer-facing commitments your team documents there.
4. Decide rollback, partial rollback, feature flag off, or continue observing. Document the decision and the metrics that drove it in the incident or change ticket.

Rollback execution stays manual or follows your existing runbook automation. The assistant does not call rollback APIs, flip traffic weights, or revert Git refs on its own. That boundary protects teams from model drift, misconfigured thresholds, and transient spikes during deploy noise.

For teams building formal procedures, fold verified metric checks into artifacts from deployment runbook generation so every rollback path lists the same signals the assistant watches. Before large changes, change impact simulator exercises which metrics are likely to move and whether thresholds are calibrated too tight or too loose.

When signals fire before user-visible impact spreads, proactive incident detection complements rollback recommendations by catching broader regressions the deploy-scoped model might not score.

Recommendations without metric cites. If output says "consider rollback due to degraded performance" but names no metric, threshold, or window, discard it for decision purposes. Fix the template or model prompt so cites are mandatory fields, not prose garnish. On-call should not rollback on vibes encoded as confidence scores.

Treating recommendation as rollback. Automation hooks, chat bots, and paging rules sometimes forward "ROLLBACK RECOMMENDED" as if rollback already happened. That creates double work and erodes trust when the service was healthy. Label messages clearly: recommendation only, human action required.

Invented SLO breach percentages. Models and templates sometimes hallucinate precise SLO figures because SLO language appears in training data. Unless your platform exports an SLO object with the same identifier the assistant references, ban SLO percent fields from output. Stick to raw or aggregated metrics your thresholds actually use.

Threshold drift without ownership. Anomaly thresholds copied from a sibling service or left unchanged after architecture shifts produce chronic false positives. Assign an owner per service to review thresholds after each major deploy pattern change.

Observation windows that do not match deploy cadence. A one-minute window on a service that needs fifteen minutes to warm caches generates noise. Align window length with how your traffic ramps and how long cold-start effects last.

Silent failure when feeds drop. If Datadog or Splunk ingestion lag exceeds the observation window, the assistant may see stale data and stay quiet while reality diverges. Monitor pipeline health with the same rigor as application metrics.

Teams on Datadog typically bind deploy markers to dashboard queries the assistant reads. Splunk-heavy shops often materialize deployment-scoped searches that emit threshold-friendly fields. GitLab supplies release metadata, environment names, and commit SHAs that anchor "which change are we evaluating." ServiceNow ties the human process together: who approved the change, which services are in scope, and where post-deploy notes land.

None of these tools replaces the others in this pattern. Observability supplies measurements, CI/CD supplies the change identity, ITSM supplies accountability. The rollback decision assistant sits in the middle as a read-only analyst that formats evidence for a human decision.

Start with one high-churn service, a short list of rollback-critical metrics, and explicit empty-state behavior when data is missing. Expand only after on-call confirms that recommendations with full cites save time compared to manual chart comparison alone.