The big idea
What is webhook ingestion?
A customer’s system fires an HTTP event at your endpoint every time something happens — a payment, a push, a status change. Sounds trivial: receive the POST, do the work. But the sender is flaky, retries on failure, and can burst; and the events are usually money, data, or state you cannot afford to lose or double-count.
Build a pipeline, not an endpoint: verify the event is authentic, acknowledge fast and buffer it in a durable queue, process idempotently, dead-letter what fails, and replay when you’ve fixed the cause. It turns an at-least-once firehose into effectively exactly-once processing.
How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.
Step 1 · The skeleton
Receive and write
The naive version: the Ingest API takes the POST and writes straight to the Target Store, synchronously, then returns. It works in a demo — and falls apart the moment the sender is untrusted, the write is slow, or anything downstream hiccups.
Stand up the Ingest API as the single front door. For now it just receives an event and writes it to the target. Everything that follows is about making this path authentic, fast, and durable.
A pipeline, not a handler: The endpoint is the easy 10%. The 90% is what happens when the sender misbehaves — which it will, because senders retry on any non-2xx.
Step 2 · Who sent this?
Verify + replay protection
Your endpoint is public, so anyone can POST to it. And a captured event could be replayed later. Trusting the payload as-is lets an attacker forge or repeat events.
Design decision: Your webhook endpoint is public. How do you know an event is genuinely from the customer and not forged or replayed?
The call: Verify an HMAC signature over the raw body, and reject stale timestamps. — Recompute the signature with the shared secret and constant-time-compare it, and reject events older than a few minutes. Only an authentic, fresh event passes — this is how Stripe/GitHub webhooks work.
Add a Verify step: recompute the HMAC signature over the raw body, constant-time-compare it to the header, and reject events outside a short replay window. Forged and repeated events are dropped at the door.
Authenticate the payload, not the pipe: Verify over the raw bytes (a re-serialized JSON won’t match), use a constant-time compare, and bound freshness with a timestamp — signature plus replay window is the standard.
Step 3 · Ack fast, work later
Queue + async worker
If the Ingest API processes the event synchronously and processing is slow — or the target is briefly down — your response is slow, so the sender times out and retries, piling on more load exactly when you’re struggling. Coupling ingestion to processing is a trap.
Design decision: Processing an event is slow and the target is sometimes down. If you do it synchronously, the sender times out and retries. What decouples them?
The call: Ack fast: enqueue the event in a durable queue, process asynchronously. — The gateway persists the event to a durable queue and returns 2xx in milliseconds; a separate worker processes at its own pace. A slow or dead worker becomes lag, not lost data — and the sender never retries for slowness.
The Ingest API writes the verified event to a Durable Queue and immediately returns 2xx. A separate Worker pulls from the queue and writes to the Target Store at its own pace. Ingestion and processing are now decoupled.
Decouple ingestion from processing: A durable queue turns a processing outage into consumer lag instead of dropped events. Ack in milliseconds so the sender is happy; drain the backlog when the worker recovers.
Step 4 · Never process twice
Idempotent processing
Senders retry on any non-2xx, and even a successful event can be re-sent after an ambiguous timeout (you processed it but your ack was lost). Delivery is at-least-once, so the same event will arrive more than once — and processing it twice double-charges or double-writes.
Design decision: Delivery is at-least-once — the same event will arrive more than once. How do you avoid processing it twice?
The call: Record each event’s id in an idempotency store; skip ids already seen. — On first processing, record the event id and the result; on any redelivery the id is already present, so you skip it (or return the recorded result). At-least-once delivery now behaves like exactly-once.
Give the Worker an Idempotency Store keyed by event id. Before processing, check the id; if it’s already there, skip it. Otherwise process, then record the id. Redeliveries become no-ops — effectively exactly-once.
Idempotency makes retries safe: You can’t stop redeliveries, so make them harmless. Dedupe on a stable id (the sender’s event id or an idempotency key), and the whole retry machinery becomes safe.
Step 5 · When it just won’t process
Retries + dead-letter
Some events fail every attempt — a malformed payload, a permanently-rejecting downstream, a bug. Retrying them forever blocks the queue (head-of-line blocking) and burns resources; dropping them silently loses data. Neither is acceptable.
Design decision: A poison event fails every retry. Retrying forever blocks the queue; dropping it loses data. What do you do with it?
The call: After N retries, move it to a Dead-Letter Queue for inspection and replay. — Bounded retries with backoff handle transient failures; anything that exhausts them goes to a DLQ — off the main path, held for a human to inspect and later replay. Silent loss becomes a visible, recoverable backlog.
The Worker retries transient failures with backoff, but after N attempts it moves the event to a Dead-Letter Queue. The main queue keeps flowing, and the failed event is parked — visible, countable, and recoverable — instead of lost.
Bounded retries, then dead-letter: Retries fix the transient; the DLQ contains the permanent. Alert on DLQ depth: a rising dead-letter count is your earliest signal that something upstream broke.
Step 6 · Fix it, then replay
Replay + monitoring
Events piled up in the DLQ during an outage or a bug. Once you’ve fixed the cause, you need those events processed — you can’t just tell the customer their data is gone. And you need to have known the pipeline was struggling before they told you.
A Replay worker re-enqueues dead-lettered events back onto the main queue once the fix is in — safe precisely because processing is idempotent, so anything already handled is skipped. Meanwhile Ops watches consumer lag and DLQ depth, with alerts, so a backup is caught in minutes, not weeks.
Recoverable + observable: A DLQ without replay is just a graveyard; replay makes it a recovery tool. And lag + DLQ-depth dashboards are how you learn the pipeline broke before the customer does — essential when you can’t see the box directly.
You did it
You just designed webhook ingestion at scale.
- An Ingest API is the single front door — verify, then ack fast.
- HMAC signature over the raw body + a replay window reject forged and repeated events.
- A durable queue decouples ingestion from processing: an outage is lag, not loss.
- An idempotency store dedupes redeliveries, making at-least-once behave like exactly-once.
- Bounded retries with backoff, then a dead-letter queue for poison messages.
- A replay worker recovers the DLQ once fixed — safe because processing is idempotent.
- Consumer-lag and DLQ-depth monitoring catch trouble before the customer does.