type: user

role: user

content: Investigate webhook delivery failures for the EU billing queue. Alerts fired at 03:10 UTC showing a 70% delivery failure rate.


type: assistant

role: assistant

content: Initial hypothesis: the EU gateway is down. Checking gateway health.


content: Gateway health is green; gateway is NOT the cause. Rejecting the gateway-down hypothesis. Next hypothesis: the delivery route is misconfigured.


content: Did you find anything in the delivery logs?


content: Delivery logs show the carrier retried duplicate deliveries for the same event id up to 8 times, exhausting the retry budget. The stale route eu-billing-legacy was still receiving traffic. Confirmed root cause: the eu-billing-legacy route was not decommissioned after the migration, causing duplicate delivery and retry exhaustion on the active route.


content: Mitigation applied: disabled the stale eu-billing-legacy route. Delivery failure rate dropped from 70% to 2% within 5 minutes. No data loss; all events eventually delivered via the active route.


content: What's still open after this?


content: Follow-up risk: there is no replay protection in the runbook, so a future stale route or carrier-side duplicate can cause the same retry exhaustion. The runbook needs an idempotency/dedup step before this can be considered resolved. Owner decision: routing decommission authority sits with the Platform team, not the on-call responder. I am leaving the runbook-gap follow-up open for Platform to add replay protection.