A demo delivers one event on a good day. A real integration delivers thousands against an endpoint that fails intermittently — and worse, sometimes processes an event but times out before it can say so. Naively, you lose events. Add retries and you stop losing them, but you start duplicating the ones that timed out. Add idempotency and the duplicates vanish. Add a dead-letter queue and the events that exhaust their retries become recoverable instead of lost. Send a stream through a flaky endpoint, toggle the three patterns, and watch the exactly-once delivery rate climb — the reliability trade-offs from the integration handbook, made playable.
A temporary error — a timeout, a 503, a network blip — that would succeed if simply tried again.
ambiguous timeout
The server processed the event but the client never got the ack, so the client thinks it failed. Retrying causes a duplicate.
idempotency
A re-delivered event is processed at most once, by deduping on a key. It’s what makes retries safe.
dead-letter queue
Where an event goes after exhausting its retries — held for inspection and replay instead of dropped.
delivery.js — retry, dedupe, dead-letter
Ready
The endpoint succeeds ~60% of the time, times out ~15% (it processed the event but you didn’t hear back), and hard-fails ~25%. Toggle the three patterns, then Send 20 and watch the outcomes. Start with everything off to feel the pain.
—
delivered once
—
duplicated
—
lost
How it works
The lab is a tiny delivery simulator, and building it exposes exactly why the three patterns exist and why their order matters. Model each attempt as one of three outcomes: a clean success (the server processes it and you get the ack), an ambiguous timeout (the server processes it but you never hear back), or a hard failure (nothing happened). Now watch the patterns interact. With no retries, every failure — hard or timeout — that you can’t confirm becomes a lost event, so your delivery rate is capped by the endpoint’s reliability. Turn on retries and losses plummet, because a hard failure just gets tried again until it sticks — but a nasty thing appears: an ambiguous timeout already processed the event, and retrying it makes the server process it again, so retries manufacture duplicates. This is the counter-intuitive lesson — retries alone trade one failure mode (loss) for another (duplication). The fix is idempotency: dedupe on a client-supplied key so a re-delivered event is processed at most once, which collapses those duplicates back to exactly-once and is precisely why idempotency is the prerequisite that makes retries safe. Finally, some events are just unlucky and exhaust every retry; without a dead-letter queue they’re silently dropped, but with one they’re parked for inspection and replay — turning an invisible data-loss bug into a visible, recoverable backlog. Stack all three and the exactly-once rate approaches 100% even against a genuinely flaky endpoint. The takeaway an FDE carries to every integration: retries need idempotency, and everything needs a dead-letter path.
1
Naive delivery loses events
With no retries, every hard failure — and every timeout you can’t confirm — is a lost event. Your delivery rate is capped by the endpoint’s reliability, which you don’t control.
2
Retries cut loss — but add duplicates
Retrying a hard failure eventually succeeds, so losses plummet. But an ambiguous timeout already processed the event; retrying it processes it again. Retries trade loss for duplication.
3
Idempotency kills the duplicates
Dedupe on a key so a re-delivered event is processed at most once. The duplicates collapse to exactly-once. This is why idempotency is the prerequisite that makes retries safe.
✓
Dead-letter recovers the unlucky
A few events exhaust every retry. Without a dead-letter queue they vanish silently; with one they’re parked for replay. Stack all three and exactly-once approaches 100% — even on a flaky endpoint.
Retries alone
less loss, more duplicates
Idempotency
makes retries safe
Dead-letter
loss → recoverable
All three
≈ exactly-once
The code
# Deliver one event through a flaky endpoint
processed = 0
for attempt in range(max_attempts if retries else 1):
r = try_deliver(event) # success | timeout | failif r == "success": processed += 1; break# got the ack — stopif r == "timeout": processed += 1 # server DID process it…# …but we saw a failure, so we retry → maybe process again
effective = min(1, processed) if idempotency else processed
if effective == 0: dead_letter(event) if dlq else drop(event)
elif effective == 1: delivered_exactly_once += 1
else: duplicated += 1 # retries without idempotency
Quick check
1. You add retries to a flaky integration and losses drop — but a new problem appears. What is it?
An ambiguous timeout means the server processed the event but the client saw a failure. Retrying it makes the server process it a second time. So retries alone trade one failure mode (loss) for another (duplication) — which is why they need idempotency.
2. What makes retries safe?
Idempotency ensures that no matter how many times an event is delivered, it’s processed at most once. That collapses the duplicates retries create back to exactly-once, which is why idempotency is the prerequisite for safe retries.
3. What does a dead-letter queue give you?
Some events fail every retry. Without a dead-letter queue they’re dropped silently — an invisible data-loss bug. A DLQ parks them for inspection and replay, turning silent loss into a visible, recoverable backlog you can act on.
FAQ
Why do retries alone cause duplicate deliveries?
Because of ambiguous timeouts: the server processes an event but the ack is lost, so the client thinks it failed and retries. The server, with no memory of the first delivery, processes it again — a duplicate. Retries fix hard failures but manufacture duplicates from timeouts. The fix is idempotency: dedupe on a client-supplied key so the second delivery is ignored.
What is idempotency and why does it matter for integrations?
An operation is idempotent if doing it many times has the same effect as doing it once. For integrations it means a re-delivered webhook or retried request is processed at most once, usually by deduping on an idempotency key: the first time you see the key you process and record the result; later times return the recorded result. It matters because retries re-deliver events, and without idempotency those re-deliveries double-charge or double-write.
What is a dead-letter queue?
A dead-letter queue (DLQ) is where a message goes after failing processing too many times. Instead of being retried forever (blocking the pipeline) or dropped silently (losing data), it’s moved to a separate queue to inspect, fix, and replay. For an FDE, a DLQ turns silent, invisible data loss into a visible, recoverable backlog you can monitor and alert on. Any reliable ingestion path should have one.
How do you make a webhook integration reliable?
Combine patterns: verify the signature and reject stale events (replay window); acknowledge fast (return 2xx) and process asynchronously so slowness doesn’t trigger retries; make processing idempotent by deduping on the event id (senders retry on non-2xx and re-deliver); and route failures to a dead-letter queue for replay. Together these turn an at-least-once channel into effectively exactly-once processing.