The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a notification system?
Across a product, dozens of services have something to tell a user: your order shipped, someone messaged you, your password changed. Each one should not have to know about Apple’s push servers, Twilio, or your email provider.
Build one service every team calls with “notify user U about event E.” It figures out the who, what and how — the right channels, the user’s preferences, the templates — and handles the messy delivery to third parties. One throat to choke, one place to get reliability right.
What the new pieces do
- Servicesproducer
- Any backend service that needs to tell a user something — an order shipped, someone liked your post, a login from a new device.
Step 1 · The front door
One API for everyone
If every service integrates Apple, Google, Twilio and an email provider on its own, you get four copies of the same fragile code — and no consistent way to throttle, template, or respect a user’s unsubscribe. How should services send?
A dozen services each need push, SMS and email. How do they integrate delivery?
Four copies of fragile provider code in every service, four places to fix every bug, and no consistent throttling, templating or unsubscribe. Provider quirks leak everywhere.
Better than nothing, but credentials, rate limits and retries still run in every caller’s process — you can’t enforce a global cap or audit centrally, and every service must redeploy to fix delivery.
Producers make one call and walk away. A single service owns channels, preferences, templates, retries and provider credentials — one place to get reliability and compliance right.
Put a single Notification API in front. A producer makes one call — POST /notify {user, type, data} — and walks away. The API owns everything downstream, so senders stay blissfully ignorant of how delivery actually happens.
Why this piece earns its place
Notice what the payload does not carry: a channel. That omission is the contract, and it is the first thing a producer will ask you to break — someone owns an urgent flow, knows it should go by SMS, and wants a channel field. Give it to them and producers start making delivery decisions again, the consent gate becomes advisory, and the copy for one channel gets written by a team that never sees the other two. Urgency is real, so model it where it belongs: as a property of the notification type — how time-sensitive it is, whether it is transactional — which this service maps to channels and to how hard it tries. The type is also the unit a user opts out of, so the same field that decides how loud a send is decides whether it is allowed at all. The day the assumption bends is a partner flow that genuinely owns its own copy and channel; answer that with a new type and its own templates, not with a bypass, or the front door stops being a front door.
What the new pieces do
- Notification APIbackend
- One front door every producer calls. It validates the request and hands it to the pipeline, so senders never talk to providers directly.
Step 2 · Don’t block the sender
Decouple with a queue
Calling Apple or Twilio can take hundreds of milliseconds and sometimes fails. If the API delivered inline, a slow provider would stall the producer’s request — and a provider outage would take the whole thing down. How should the API respond fast?
Calling Apple/Twilio takes ~100s of ms and sometimes fails. How does the API stay fast?
Now the producer’s request is hostage to the slowest provider, and a provider outage stalls or fails every caller. Synchronous delivery couples your latency to third parties you don’t control.
The API does a fast local write to a durable queue and returns; a worker fleet drains it at provider speed. Outages and slowness become a backlog that waits, not lost requests.
Unbounded threads vanish on a crash (losing in-flight sends), can’t apply backpressure, and leave no durable record. A queue gives durability, retries and flow control for free.
The API just validates and enqueues, then returns instantly. A pool of delivery workers drains the queue (Kafka) and does the slow work. Producer latency is now a fast local write; delivery happens asynchronously.
Why this piece earns its place
The instant response means accepted, and every caller will read it as delivered. Under normal load the difference is invisible; during a provider outage it is the whole story, because the queue does exactly what it promised and holds everything, and a password reset surfaces from the backlog long after the user gave up and asked for another. Depth is the wrong alarm — a queue can be shallow and still old. Watch the age of the oldest unprocessed message, per class of traffic. And keep the classes apart: a digest run and a 2FA code must not share a lane, because one may wait hours and the other is worthless in minutes. Separate topics or partitions with their own worker pools give the urgent lane a floor that bulk traffic cannot eat into. The other consequence of returning early is that the producer now knows nothing about the outcome — if a screen has to say the message was sent, that state comes back from tracking later, never from the enqueue.
- <10msenqueue
- 1000s/singest
- asyncdelivery
What the new pieces do
- Delivery Workersworker
- Drain the queue, render the template, pick the channel and call the provider. A stateless fleet you scale to match send volume.
- Notification Queuebus
- Decouples the fast API from slow third-party delivery. The API drops a message and returns; workers drain the queue at their own pace.
Back of the envelope
- enqueue ≈ <10ms local write
- vs ~100s of ms per third-party provider call
- 1 API tier ⇒ 1000s of msgs/s
- workers scale out independently to match delivery rate
- outage ⇒ backlog in the log
- durable wait, zero loss — drained when capacity returns
Step 3 · Respect the human
Did they even want this?
Blasting every event to every channel is how you get users to mute you — or report you as spam, which wrecks your sender reputation. People want control: which categories, which channels, and not at 3am. How do you respect that?
How do you avoid spamming users into muting you or reporting spam?
Pushing OS-mute onto users tanks engagement and your sender reputation, and ignores quiet hours and legal opt-out requirements. Consent has to live in your system.
A global rule ignores that each user wants different things, and preferences change. Per-user, per-type, per-channel consent belongs in data, not code.
Every notification passes a consent gate that honors opt-outs, channel choice and quiet hours. It’s both good manners and the law (CAN-SPAM, GDPR).
Before sending, a worker checks the Preference Service: is this user opted in to this type, on this channel, right now? It enforces quiet hours and unsubscribes, dropping or deferring anything the user didn’t ask for.
Why this piece earns its place
This plants a synchronous dependency on the hot path — every send asks it a question — so the real decision here is what a worker does when the Preference Service will not answer. Both defaults are bad in opposite directions. Fail open and you send to people who unsubscribed, which is the one mistake on this page you cannot take back and the one with a regulator attached. Fail closed and you drop password resets and 2FA codes during an outage. So do not choose globally: types a user cannot opt out of never consult preferences at all, and everything opt-outable fails closed, waits in the queue, and is re-checked when the service returns. Underneath that, keep a per-user snapshot in the workers, refreshed from a change stream, so the common path is not a live call per send. That cache gets one hard rule: an unsubscribe invalidates immediately, a fresh opt-in may arrive lazily. Being slow in that direction only ever costs a send.
What the new pieces do
- Preference Serviceservice
- Decides whether this user wants this kind of notification, on which channels, and whether it is their quiet hours. The unsubscribe gatekeeper.
Step 4 · Many shapes, many channels
Render once, fan out
“Order shipped” must become a 20-character push title, a 160-character SMS, and a full HTML email — in the user’s language. And it should go to every channel they’ve enabled, not just one.
Workers pull from a Template Store (variables + localization) to render the right copy per channel, then fan out to Push (APNs/FCM), SMS (Twilio) and Email (SES). Each channel is an adapter that speaks one provider’s protocol.
Why this piece earns its place
Templates are data, so copy now ships on its own path — no deploy, no review from the people who own the pipeline. That is the point of the store, and it is also how this step breaks: one wrong variable name renders literally, to everyone the event fans out to, on every channel at once, so a typo lands three times per person. Treat publishing a template like a schema change — declare the variables each notification type provides, validate a template against that list before it can go live, version it, and render a test send first. The second trap is length. A 20-character push title and a 160-character SMS are not one sentence shortened; truncating a rendered string is how a translation loses its verb, and the languages that run longest are exactly the ones you would cut. Author copy per channel per locale and let the store hold them all under one template id — the source stays single without pretending every channel says the same words.
- 3+channels
- Nlanguages
- 1template
What the new pieces do
- Template Storestore
- Holds message templates with variables and localization, so a worker can render “Hi {name}, your order shipped” in any language.
- Pushchannel
- Delivers to phones via Apple (APNs) and Google (FCM). The most common channel and the most latency-sensitive.
- SMSchannel
- Short text to a phone number through a provider like Twilio. Expensive per message, so reserved for high-value alerts.
- Emailchannel
- Rich, cheap, and tolerant of delay. Handled by a transactional provider that manages reputation, bounces and spam folders.
Back of the envelope
- 1 event × channels enabled
- one notify fans out to push + SMS + email
- 1 template × N languages
- rendered per channel/locale from a single source
- new channel = new adapter
- Slack/WhatsApp added without touching any producer
Step 5 · Not all events are live
Scheduled & triggered sends
Plenty of notifications aren’t reactions to an event: a daily digest, a “your trial ends tomorrow” reminder, a re-engagement nudge after 7 days of silence. Nothing fires these unless something is watching the clock. How do you send them?
Digests and "trial ends tomorrow" reminders aren’t event reactions. How are they sent?
Time-based sends funnel through the identical pipeline — preferences, templates, channels, tracking — so a scheduled digest behaves exactly like a live alert. One path, uniform behavior.
Now consent, templating, tracking and rate limits exist twice and drift apart. A new source of notifications shouldn’t mean a new delivery path.
Scattering schedules across every service means no central view, duplicated cron logic, and inconsistent throttling. Centralize the clock the way you centralized delivery.
Add a Scheduler that emits notifications at the right time, calling the same Notification API as everything else. Reuse the entire pipeline — preferences, templates, channels — so scheduled sends behave identically to live ones.
Why this piece earns its place
A scheduled send quietly assumes the thing it announces is still true when the clock fires, and time guarantees the opposite. “Your trial ends tomorrow,” queued the night before, is wrong if the user upgraded at breakfast; a digest assembled when it was scheduled is a list of things that may since have been deleted. So have the Scheduler emit an intent — this user, this type, this window — and let the worker resolve the content and re-check the condition at render time, right beside the preference check it already makes. Late binding costs one read per send and buys correctness on everything that can change underneath it. The other half is cancellation: whatever was scheduled must be removable by the same key it was scheduled under, so upgrading the plan deletes the reminder instead of racing it. And because a cron fires per window rather than per event, the failure to design for is a missed run, not a slow one — a skipped window is silent, so record that each window was handled and be able to replay one that was not.
What the new pieces do
- Schedulertrigger
- Fires time-based notifications — daily digests, reminders, drip campaigns — on a schedule rather than in reaction to a live event.
Step 6 · Did it actually arrive?
Track delivery & retry
Providers fail in a dozen ways: a device token expires, an email bounces, Twilio rate-limits you. Fire-and-forget means you never learn — and the user silently misses something important. How do you make delivery reliable?
Providers fail silently — expired tokens, bounces, rate limits. How do you not lose sends?
You never learn about failures, so users silently miss important alerts and you can’t debug "why didn’t I get it?". Delivery must be a measured fact, not a hope.
Permanent failures (dead token, hard bounce) never succeed — retrying them forever burns provider quota for nothing. You must distinguish transient from permanent.
Delivery Tracking makes every send auditable; transient failures retry with backoff (idempotently, so no dupes), and permanent ones are flagged to stop trying and fall back to another channel.
Every attempt writes its outcome to Delivery Tracking — sent, delivered, bounced, opened. Failures get retried with backoff; permanent ones (dead token, hard bounce) are marked so you stop trying and can fall back to another channel.
Why this piece earns its place
Three tries with backoff is a starting point, not a policy, because the count is the wrong knob — the deadline is. Every notification has a shelf life: a 2FA code is worthless before the third attempt lands, an order-shipped mail is fine arriving hours late, a flash-sale alert is embarrassing once the sale ends. Give each type an expiry, let the retry schedule run until it, then stop and record the outcome as expired rather than failed, because those are different bugs and only one of them is the provider’s. Backoff without a ceiling is the failure at the other end: a provider that rate-limits you gets retried by every worker at once, the backlog grows, and your retries have become the outage — so jitter the delays and cap attempts in flight per provider. And the transient-versus-permanent split rests on the provider’s own error codes, which lie in one specific way: a provider that delivered and then failed to answer reports the same thing as one that never delivered. That is why retries only work paired with a stable key.
- 99.9%delivered
- ×3retry w/ backoff
- bouncefeedback loop
What the new pieces do
- Delivery Trackingstore
- Records every send’s outcome — delivered, bounced, opened — feeding retries, analytics, and the “why didn’t I get it?” debugging.
Back of the envelope
- transient fail ⇒ retry ×3 backoff
- rate-limited / token-expired recover on their own
- hard bounce ⇒ stop + fall back
- don’t burn provider quota on permanent failures
- every attempt ⇒ tracked outcome
- delivery becomes measurable, debuggable, auditable
Step 7 · The sharp edges
No duplicates, no floods
Retries, multiple producers, and at-least-once queues all conspire to send the same alert twice. And a buggy loop or a viral post can fan out thousands of notifications to one poor user in a minute. How do you tame both?
At-least-once queues can double-send, and a viral post can fire 500 alerts in a minute. Fix?
Users experience that as broken spam. At-least-once is unavoidable across third parties, but you can collapse it to effectively-once and cap floods — so you should.
A stable key collapses duplicate sends to one; per-user limits and rollups turn "🔥 liked your post" ×500 into "500 people liked your post." Effectively-once and flood-proof.
Exactly-once across external providers is a fantasy — the provider may deliver, then fail to ack. Aim for at-least-once + dedup on a stable key, which actually works.
Add a Dedup + Rate Limit guard. An idempotency key per logical notification collapses duplicates; per-user, per-type rate limits (and digest-rollups) cap the flood so “🔥 liked your post” ×500 becomes “500 people liked your post.”
Why this piece earns its place
Everything here rides on the key, and its granularity is the design. Too fine — a timestamp, a message id, an attempt number — and it never collides, so duplicates flow through and the guard is decoration. Too coarse — user plus type — and two different orders shipping on the same day collapse into one notification the user never gets. Derive the key from the triggering thing itself: the user, the type, and the entity or time window the event is about. Coarse is the dangerous direction, because a notification that was never sent raises no error and appears in no failure count — so write suppressions into Delivery Tracking with their reason and read that number the way you read bounces. Limits need the same honesty about what capped means: dropping is defensible only where a rollup catches what was dropped; without one, hitting a limit has to defer the send rather than discard it. And this guard is the one place a hot key is guaranteed — the viral post is by definition many increments of a single user’s counter in the same second, so count approximately per window rather than exactly and slowly.
What the new pieces do
- Dedup + Rate Limitguard
- Stops the same alert firing five times and caps how many notifications a user gets per hour, using idempotency keys.
You did it
You just designed a notification system.
Everything you assembled, in order
- One Notification API as the single front door for every producer.
- A queue + worker fleet so slow third-party delivery never blocks senders.
- A Preference Service enforcing opt-ins, channels and quiet hours.
- Template-driven rendering that fans out to push, SMS and email.
- A scheduler reusing the same pipeline for digests and reminders.
- Delivery tracking with retries and bounce feedback for reliability.
- Idempotency-key dedup and per-user rate limits to kill duplicates and floods.