The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is webhook ingestion?
A customer’s system fires an HTTP event at your endpoint every time something happens — a payment, a push, a status change. Sounds trivial: receive the POST, do the work. But the sender is flaky, retries on failure, and can burst; and the events are usually money, data, or state you cannot afford to lose or double-count.
Build a pipeline, not an endpoint: verify the event is authentic, acknowledge fast and buffer it in a durable queue, process idempotently, dead-letter what fails, and replay when you’ve fixed the cause. It turns an at-least-once firehose into effectively exactly-once processing.
Step 1 · The skeleton
Receive and write
The naive version: the Ingest API takes the POST and writes straight to the Target Store, synchronously, then returns. It works in a demo — and falls apart the moment the sender is untrusted, the write is slow, or anything downstream hiccups.
Stand up the Ingest API as the single front door. For now it just receives an event and writes it to the target. Everything that follows is about making this path authentic, fast, and durable.
Why this piece earns its place
The quiet assumption here is that one public URL, one handler and one secret serve everybody. That holds exactly until the second sender arrives, and then the questions you get paged about have no answer: whose traffic just tripled, whose secret needs rotating, whose malformed payloads are filling the logs. None of that is recoverable after the fact — it has to be in the request. So carry the sender’s identity in the path or a header from day one, even with a single customer, and key everything operational off it: a secret per sender, a rate limit per sender, and a dashboard you can slice by sender. Version the path while you are at it. A sender will change its payload shape eventually, and one shared endpoint means you either break everyone at once or branch on the body — which is how one customer’s odd event becomes an outage for the rest.
What the new pieces do
- Senderclient
- The customer’s system (Stripe, GitHub, their own app) POSTs an event to your endpoint whenever something happens. It retries on any non-2xx, so it will re-deliver.
- Ingest APIbackend
- The single front door. Verifies the signature, checks the idempotency key, enqueues the event, and returns 2xx — all in milliseconds, so the sender never retries for slowness.
Step 2 · Who sent this?
Verify + replay protection
Your endpoint is public, so anyone can POST to it. And a captured event could be replayed later. Trusting the payload as-is lets an attacker forge or repeat events.
Your webhook endpoint is public. How do you know an event is genuinely from the customer and not forged or replayed?
IPs are spoofable and senders’ IP ranges change; allow-lists break and don’t prove the payload wasn’t tampered with. You need to verify the content, not just the origin.
TLS protects the data in transit but proves nothing about who sent it — anyone can open an HTTPS connection to your public endpoint and POST a forged event.
Recompute the signature with the shared secret and constant-time-compare it, and reject events older than a few minutes. Only an authentic, fresh event passes — this is how Stripe/GitHub webhooks work.
Add a Verify step: recompute the HMAC signature over the raw body, constant-time-compare it to the header, and reject events outside a short replay window. Forged and repeated events are dropped at the door.
Why this piece earns its place
The replay window is the knob this step introduces, and what it really has to cover is narrower than it looks: honest clock skew between the sender’s host and yours, and nothing else. A well-behaved sender re-signs every delivery attempt, so a retry that lands an hour later arrives with a fresh timestamp and a fresh signature — it is not a stale event, and the window is not a budget for the sender’s backoff. Check that per sender before assuming it: one that signs once and reuses that signature across attempts is the exception, and it drags the window out to the sender’s whole retry horizon. That leaves the window as a pure security knob. Widen it and a captured request stays usable for that much longer; the window bounds how long a theft is worth anything, it never makes a replay harmless by itself. Tighten it and ordinary skew starts rejecting real events, which you will read in the logs as forgeries. A few minutes is the usual compromise, and you make it survivable by keeping your hosts on NTP and treating a spike in stale-timestamp rejections as a clock alarm before a security one. Rotation needs the same care: accept the old and new secret together for an overlap and try both, or a routine rotation reads exactly like an attack.
- HMACover raw body
- ~5 minreplay window
- const-timecompare
What the new pieces do
- Verifyservice
- Recomputes the signature over the raw body and constant-time-compares it, and rejects events older than a few minutes — so only authentic, fresh events get in.
Back of the envelope
- signature over raw body
- re-serialized JSON breaks it
- constant-time compare
- no timing side-channel
- reject old timestamps
- stops replay attacks
Step 3 · Ack fast, work later
Queue + async worker
If the Ingest API processes the event synchronously and processing is slow — or the target is briefly down — your response is slow, so the sender times out and retries, piling on more load exactly when you’re struggling. Coupling ingestion to processing is a trap.
Processing an event is slow and the target is sometimes down. If you do it synchronously, the sender times out and retries. What decouples them?
More servers don’t fix slow synchronous processing — each request still blocks on the same downstream work, so senders still time out and retry. You need to stop blocking the response on processing.
The gateway persists the event to a durable queue and returns 2xx in milliseconds; a separate worker processes at its own pace. A slow or dead worker becomes lag, not lost data — and the sender never retries for slowness.
You don’t control the sender’s retry policy, and slowing retries doesn’t make your processing durable. The fix is on your side: acknowledge quickly and buffer the work.
The Ingest API writes the verified event to a Durable Queue and immediately returns 2xx. A separate Worker pulls from the queue and writes to the Target Store at its own pace. Ingestion and processing are now decoupled.
Why this piece earns its place
The moment you return 2xx before the work is done, that status code stops being a receipt and becomes a promise: the sender will never send this event again, and nothing downstream can ask it to. So the only thing allowed to happen before the ack is a write the queue has genuinely made durable — replicated or fsynced and acknowledged by the broker, not buffered in the gateway’s memory to be flushed a moment later. A gateway that dies holding a few thousand in-flight events loses them silently, and the one party who could have re-sent them has already been told they arrived. The corollary is the part people skip: when the queue is unavailable, the correct response is a failure. This is the one place in the design where deliberately failing the request is the safe move, because a non-2xx hands the event back to the sender’s retry machinery — a durable buffer you did not have to build, operate or pay for.
- <50msack
- durablebuffer
- lag ≠ losson outage
What the new pieces do
- Durable Queuecache
- A durable log (Kafka/SQS) holding accepted events. It decouples ingestion from processing, so a slow or dead worker becomes lag, not lost data — and absorbs bursts.
- Workerservice
- Pulls events off the queue and does the real work (write a record, call an API). Retries transient failures with backoff; after N tries it sends the event to the DLQ.
- Target Storedata
- Where processed events land — a database, a data warehouse, another API. The end of the pipeline, and the thing that must never see a duplicate.
Back of the envelope
- ack fast (2xx in ms)
- sender never retries for slowness
- say 5k events/s × a 30-min downstream outage = 9M events
- the queue absorbs every one of them — this is what “lag ≠ loss” costs
- 9M × ~2 KB ≈ 18 GB of buffered events
- size the queue for the outage you expect to survive, not for the steady state
- durable queue
- buffers bursts, survives worker death
- async worker
- processes at its own pace
Step 4 · Never process twice
Idempotent processing
Senders retry on any non-2xx, and even a successful event can be re-sent after an ambiguous timeout (you processed it but your ack was lost). Delivery is at-least-once, so the same event will arrive more than once — and processing it twice double-charges or double-writes.
Delivery is at-least-once — the same event will arrive more than once. How do you avoid processing it twice?
On first processing, record the event id and the result; on any redelivery the id is already present, so you skip it (or return the recorded result). At-least-once delivery now behaves like exactly-once.
Exactly-once delivery over a network is effectively impossible; every real webhook system is at-least-once. The receiver has to make processing idempotent — you can’t push this to the sender.
Comparing whole payloads is slow, brittle (fields reorder, timestamps differ), and misses legitimately-identical events. A stable id (the sender’s event id or your idempotency key) is the reliable dedupe key.
Give the Worker an Idempotency Store keyed by event id. Before processing, check the id; if it’s already there, skip it. Otherwise process, then record the id. Redeliveries become no-ops — effectively exactly-once.
Why this piece earns its place
Check the store, then process, then record: written that way there is a window between the check and the record where a second worker can run the same check. Redeliveries are exactly what fills that window, because a sender retrying after an ambiguous timeout often arrives while the first copy is still in flight, so two workers each see an absent id and each do the work. The record and the effect have to commit together — a unique constraint on the event id inside the same transaction as the write, or an insert-first claim that marks the id in-progress so the second worker waits or drops. When the effect lands somewhere you cannot transact with, write the claim first and pass the same id downstream as its idempotency key. The other decision is how long ids live. They cannot be kept forever, and the floor is the sender’s longest retry horizon: expire them sooner and a late redelivery is applied a second time with nothing anywhere recording that it happened.
- at-least-oncedelivery
- dedupe by idnot payload
- ≈ exactly-onceeffect
What the new pieces do
- Idempotency Storedata
- Records the id of every event processed. On a redelivery the id is already there, so the event is skipped — making at-least-once delivery behave like exactly-once.
Back of the envelope
- store the event id after processing
- redelivery finds it and skips
- key on a stable id
- the sender’s event id / idempotency key
- result cache optional
- return the first result on replay
Step 5 · When it just won’t process
Retries + dead-letter
Some events fail every attempt — a malformed payload, a permanently-rejecting downstream, a bug. Retrying them forever blocks the queue (head-of-line blocking) and burns resources; dropping them silently loses data. Neither is acceptable.
A poison event fails every retry. Retrying forever blocks the queue; dropping it loses data. What do you do with it?
A permanently-failing event retried forever blocks everything behind it (head-of-line blocking) and never succeeds. You need to get it out of the main flow after a bounded number of tries.
Dropping on failure loses data silently — the worst kind of bug, invisible until a report is wrong weeks later. Failed events must be captured, not discarded.
Bounded retries with backoff handle transient failures; anything that exhausts them goes to a DLQ — off the main path, held for a human to inspect and later replay. Silent loss becomes a visible, recoverable backlog.
The Worker retries transient failures with backoff, but after N attempts it moves the event to a Dead-Letter Queue. The main queue keeps flowing, and the failed event is parked — visible, countable, and recoverable — instead of lost.
Why this piece earns its place
N and the backoff schedule only mean something once you know which failure you are absorbing, and the two kinds want opposite treatment. A per-event failure — an unparseable payload, a field the target refuses — fails identically on every attempt, so retries buy nothing and the event should reach the dead-letter queue almost immediately. A dependency failure is the reverse: the event is fine and the downstream is gone, so every event behind it will also exhaust its attempts, and a schedule tuned for poison messages quietly shovels the whole stream into the DLQ. What you have then is not a backlog of poison to inspect; it is your outage copied into a second queue that you now have to drain by hand, in order, on top of live traffic. So branch on the failure: retry the transient with backoff and jitter, dead-letter the permanent fast, and when the failure looks like the dependency itself, stop consuming and let lag build rather than burning attempts against something that is not listening.
- N + backoffthen DLQ
- nohead-of-line block
- alert onDLQ depth
What the new pieces do
- Dead-Letter Queuecache
- Where an event goes after exhausting its retries. Held for inspection and replay instead of dropped — turning silent data loss into a visible, recoverable backlog.
Back of the envelope
- exponential backoff + jitter
- don’t stampede a recovering downstream
- cap attempts, then DLQ
- poison messages leave the main flow
- DLQ depth is an alarm
- earliest signal of an upstream break
Step 6 · Fix it, then replay
Replay + monitoring
Events piled up in the DLQ during an outage or a bug. Once you’ve fixed the cause, you need those events processed — you can’t just tell the customer their data is gone. And you need to have known the pipeline was struggling before they told you.
A Replay worker re-enqueues dead-lettered events back onto the main queue once the fix is in — safe precisely because processing is idempotent, so anything already handled is skipped. Meanwhile Ops watches consumer lag and DLQ depth, with alerts, so a backup is caught in minutes, not weeks.
Why this piece earns its place
Replay leans on idempotency, and idempotency only promises that the same event is not applied twice. It says nothing about the order of different events, which is precisely what replay disturbs: dead-lettered events re-enter the queue behind whatever is arriving now, so an hour-old update to a record can land after the newer update that already superseded it, and the data ends up right in your logs and wrong in the target. Where events mutate shared state, apply them by the sender’s own event timestamp or version and discard anything older than what you already hold, instead of trusting arrival order. Rate-limit the replay too, because a DLQ drained flat-out competes with live ingestion for the same workers and turns a recovery into a second incident. And one note on the dashboards: lag counted in messages is not actionable on its own. Alert on projected time to drain — backlog over current rate — because that is the number that tells you whether to wait or add workers.
- replayDLQ → queue
- idempotentso replay is safe
- watchlag + DLQ
What the new pieces do
- On-call / Opsclient
- You. Watches consumer lag and the dead-letter queue, and triggers a replay once a bad batch is fixed. On a locked-down deployment this dashboard is how you know the pipeline broke.
- Replayservice
- Once the underlying bug is fixed, re-enqueues the dead-lettered events so they flow through again. Idempotency makes replay safe — already-processed ids are skipped.
Back of the envelope
- replay re-enqueues from the DLQ
- idempotency skips the already-done
- monitor consumer lag
- the earliest sign of a slow/stuck worker
- alert on DLQ depth
- know before the customer tells you
You did it
You just designed webhook ingestion at scale.
Everything you assembled, in order
- An Ingest API is the single front door — verify, then ack fast.
- HMAC signature over the raw body + a replay window reject forged and repeated events.
- A durable queue decouples ingestion from processing: an outage is lag, not loss.
- An idempotency store dedupes redeliveries, making at-least-once behave like exactly-once.
- Bounded retries with backoff, then a dead-letter queue for poison messages.
- A replay worker recovers the DLQ once fixed — safe because processing is idempotent.
- Consumer-lag and DLQ-depth monitoring catch trouble before the customer does.