Delivery failure detection and retry

Delivery failure detection and retry is the control loop that notices when a packaged deliverable never reaches its intended recipient, decides whether the failure is recoverable, and either retries automatically or escalates to an operator. It sits after packaging and routing: the package may be correct and the recipient list may be right, yet the send still fails because of a temporary mailbox issue, a rate limit, a gateway timeout, or a permanent address problem.

The audience is an operations operator responsible for recovering failed deliveries. The goal is not to invent a new delivery channel. The goal is to keep failed work visible, actionable, and time-bounded so undelivered items do not silently age out of the queue.

Related reading: Automated deliverable packaging, Multi-recipient routing logic, and Personalized delivery communication.

The monitor only runs when delivery status events are available. Typical inputs include a delivery attempt identifier, recipient address or endpoint, channel (email, webhook, portal drop, or similar), timestamp of the attempt, and a provider or gateway status code with a human-readable reason when one exists. Useful extras are attempt count, last successful delivery for that recipient, packaging job id, and any correlation id that ties the attempt back to the source work item.

Without status events, there is nothing trustworthy to classify. In that case the use case produces empty output: no retry queue entries, no escalation tickets, and no “assumed failed” rows. Guessing from missing acknowledgements alone is a different control and should not be mixed into this path.

Operators should also define a short policy set before turning the monitor on: which status codes are soft failures eligible for retry, which are hard failures that must escalate, maximum attempts, backoff schedule, and quiet hours if retries must wait for business time. Policy belongs next to the monitor, not buried in ad-hoc scripts, so every operator sees the same rules.

On each status event, the monitor maps the provider signal into a small set of internal outcomes: delivered, soft failure, hard failure, or unknown. Soft failures are temporary conditions such as mailbox full, greylisting, transient network errors, or short-lived rate limits. Hard failures are permanent or staff-only conditions such as invalid address, blocked domain, policy rejection, or explicit unsubscribe or bounce that forbids further automatic sends.

Unknown statuses should not be auto-retried. They escalate with the raw payload so a person can decide whether to treat them as soft, hard, or a monitoring gap. That keeps the retry path conservative and reduces the chance of hammering a recipient or channel that is already rejecting traffic.

Classification should be idempotent. Duplicate status events for the same attempt must not create duplicate retries or duplicate escalations. Use the delivery attempt id (or an equivalent unique key) as the dedupe key, and store the latest classification with the attempt count so operators can audit what the monitor decided and when.

For soft failures, the monitor enqueues a retry under the configured backoff. Each retry should re-use the same packaged artifact and the same routing decision unless packaging or routing was itself the failure cause. Changing content or recipients mid-retry without an explicit rule creates audit gaps and makes root-cause analysis harder.

Retries stop when delivery succeeds, when the attempt budget is exhausted, or when a later status reclassifies the case as hard. Exhausted soft failures escalate with full history: original attempt, each retry timestamp, status codes, and the packaged deliverable reference. Escalation is not a dead letter dump. It is a work item an operator can act on.

Hard bounces stay with staff. The monitor must not keep sending into a known bad address. Operator actions typically include correcting the recipient record, choosing an alternate channel, confirming suppression, or closing the work item as undeliverable with a recorded reason. The monitor’s job on hard failures is to surface the case quickly with enough context that the operator does not have to reconstruct the trail from logs.

Human-in-the-loop means the system retries or escalates; people handle permanent failures and exceptions. Automation owns the boring, repeatable recovery path. Staff own judgment calls that change master data, customer communication, or compliance posture.

A practical daily view for operators is a three-bucket board: in retry, awaiting operator, and recently recovered. “In retry” shows soft failures still inside the attempt budget, with next scheduled attempt and remaining attempts. “Awaiting operator” shows hard failures and exhausted soft failures, sorted by age so nothing sits unnoticed. “Recently recovered” confirms that retries are actually converting, which is the health signal for the policy itself.

When volume spikes, resist widening auto-retry without checking classification quality first. A sudden rise in soft failures may mean a provider outage; a rise in hard failures may mean a bad recipient list from upstream routing. Tie spikes back to Multi-recipient routing logic and packaging quality from Automated deliverable packaging before changing backoff alone.

Communication after a successful retry should follow the same personalization rules as the original send. See Personalized delivery communication. Do not announce every failed attempt to the recipient unless policy requires it. Most soft failures recover silently; noisy failure notices create more support load than they prevent.

Success looks like short time-to-detect, high share of soft failures resolved without human touch, and hard failures reaching an operator with complete attempt history. Empty output when status events are missing is correct behavior, not a silent outage of the monitor. Distinguish “no events received” (instrumentation problem) from “events received, nothing failed” (healthy day). Instrumentation gaps belong on an ops alert, not on the retry board.

Common failure modes include retrying hard bounces because status mapping is too loose; escalating everything because mapping is too strict; losing correlation ids so escalations cannot find the package; and double-sending when dedupe is weak. Another frequent miss is treating webhook timeouts as hard failures when they are usually soft. Review a sample of escalations weekly, adjust the status map, and keep the attempt budget tight enough that bad policy fails fast into human review.

Close the loop by writing the operator’s resolution back onto the recipient or endpoint record when the fix is durable (corrected address, alternate channel, permanent suppression). That prevents the same hard bounce from re-entering the retry path on the next delivery cycle.