Reliable Webhook Delivery

You need to notify customer endpoints of events reliably, even though those endpoints are flaky, slow, or occasionally down — without hammering them or losing events.

Events are enqueued for delivery. A delivery worker POSTs to the customer URL, signs the payload (HMAC) so the receiver can verify authenticity, and records the attempt. Failures retry with exponential backoff over hours. Each customer sees a delivery log and can replay failed events. Payloads carry a monotonic event ID so consumers can dedupe and order.

• At-least-once delivery + consumer-side dedupe on event ID.
• HMAC signing with a per-customer secret and a timestamp to prevent replay.
• Backoff schedule (e.g. seconds → minutes → hours) with a cutoff, then mark failed.
• Expose a replay endpoint rather than silently dropping after the cutoff.

A slow consumer ties up workers — use per-endpoint timeouts and concurrency caps. Retrying a non-idempotent consumer double-processes — that's why you sign and number events. Delivering out of order confuses stateful consumers; include a sequence number.

Shard delivery workers by customer so one bad endpoint can't starve others. Add per-endpoint circuit breakers. Offer a pull-based feed (or queue) as an alternative for high-volume consumers.