Post-Deploy Anomaly Detection

A green CI pipeline and a successful deploy do not prove the release is healthy in production. Regressions often show up minutes later as rising error rates, slower p99 latency, or a quiet drop in a business metric that no unit test covered. For an SRE on post-deploy watch, the job is to notice that shift early enough to limit blast radius without rolling back every noisy blip.

Post-deploy anomaly detection uses ML over the telemetry you already collect after a release. Systems such as Datadog Watchdog or Honeycomb can learn what “normal” looks like for a service and surface deviations that static thresholds miss. The model’s job is to flag candidates and attach enough context for a human decision. The SRE still owns whether traffic reverts.

This page sits next to release work you already do elsewhere: readiness checks before ship in Release Readiness Gate, controlled exposure via Feature Flag Configuration Advisor, and the record of what changed in Automated Changelog Generator. Anomaly detection closes the loop after the binary is live.

The detector should refuse to invent confidence from a partial picture. Treat three signal families as required inputs for a post-deploy window:

1. Error rates – HTTP 5xx, gRPC failure codes, exception rates, and client-visible error budgets, sliced by service, endpoint, and deploy version when you have them.
2. Latency – at least p50/p95/p99 (or equivalent histograms) for the paths touched by the release, compared against the pre-deploy baseline for the same traffic shape.
3. Business metrics – conversion, checkout completion, successful job throughput, or whatever KPI proves users still get the outcome the change was meant to protect.

If any of those three is missing, empty, or not yet populated for the watch window, the system must return empty output: no anomaly score, no “probably fine,” and no rollback recommendation. A model that scores health from errors alone will miss slow, correct responses that destroy conversion. A model that only watches latency will miss cheap failures that still break users. Skipping business metrics turns SRE into a pure infrastructure dashboard when product impact is what matters.

Also require a clear deploy marker (version, canary step, or feature-flag cohort) so anomalies attach to a release, not to unrelated traffic. Without that join key, suppress output the same way you would for missing metrics.

Once error, latency, and business metrics are present for the post-deploy window, the model compares current behavior to a recent baseline for the same service and traffic mix. Useful flags share a few properties:

• Multi-signal agreement – prefer cases where at least two families move together (for example error rate and a business KPI), or where one family moves sharply against a stable baseline with clear deploy attribution.
• Version or cohort attribution – show whether the deviation concentrates on the new build, a canary slice, or a flag treatment, not only on the service aggregate.
• Explainable deltas – return the metric names, magnitude, start time relative to deploy, and the nearest related change from the changelog when available.
• Severity, not theater – distinguish “investigate now” from “watch for N more minutes,” without pretending the model knows your rollback policy.

Tools like Datadog Watchdog and Honeycomb already specialize in unsupervised or semi-supervised deviation detection on high-cardinality telemetry. Whatever stack you use, keep the contract the same: the detector proposes; it does not close the incident.

Auto-rollback without on-call confirm is unsafe. False positives from traffic shifts, dependency incidents, or regional outages can undo a good release and create a worse outage. Encode a hard gate:

1. Model emits a rollback signal (candidate) with evidence.
2. On-call SRE reviews the evidence, correlates with recent changes, and checks whether the blast radius is deploy-scoped.
3. Only after explicit confirm does automation execute rollback, canary halt, or flag kill-switch.

The confirm step can be a chat approval, a pager acknowledgment with a typed reason, or a protected CI job. What matters is that the model never flips production traffic on its own. SRE may still choose a softer response: freeze further rollouts, tighten a flag percentage, or open an incident without reverting.

When the SRE declines, feed that outcome back as a labeled false positive where your tooling allows it. That feedback improves baselines more safely than silent auto-revert ever will.

Define a fixed watch window after each production deploy (and after each major canary step). During that window:

• Stream the three required metric families at a cadence fine enough to catch fast regressions.
• Keep the deploy annotation and related changelog entries in the same view the SRE uses.
• Page only on confirmed anomaly flags that meet your severity bar; do not page on empty output.
• If metrics are delayed or collectors are down, treat the watch as incomplete: empty anomaly output plus a separate ops alert that monitoring itself is broken.

Outside the window, hand off to ordinary SLO burn and alert policies. Post-deploy detection is a release-time amplifier, not a replacement for day-two observability.

Pair this with feature-flag discipline when the release is gated: an anomaly on a 5% treatment should often kill the flag rather than roll back the entire fleet. Use Feature Flag Configuration Advisor so exposure plans and rollback paths stay aligned before you need them under pressure.

Common failure modes are predictable. Baseline pollution after a prior bad release teaches the model that broken is normal. Missing business metrics push teams to “green” on latency while revenue falls. Over-eager automation rolls back during a shared dependency outage and doubles the incident. Sparse traffic on low-volume services produces unstable rates that look like anomalies.

Mitigations stay operational, not magical: require the three metric families; return empty when any are absent; keep humans on the rollback path; shorten the learn window after major incidents; and always attribute to deploy version or flag cohort. Use the changelog link from Automated Changelog Generator so the SRE can answer “what just shipped?” without leaving the alert.

Done well, post-deploy anomaly detection is a precise paging and evidence system for release quality. It shortens time-to-suspicion after a ship, and it leaves the irreversible decision where it belongs: with the on-call engineer who confirms rollback.