The whole design, in writing
Learn system design by building a multi-channel notification system step by step. An interactive guide covering fan-out, message queues, user preferences, push/SMS/email channels, delivery tracking, dedup and rate limiting.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a notification system?
Across a product, dozens of services have something to tell a user: your order shipped, someone messaged you, your password changed. Each one should not have to know about Apple’s push servers, Twilio, or your email provider.
Build one service every team calls with “notify user U about event E.” It figures out the who, what and how — the right channels, the user’s preferences, the templates — and handles the messy delivery to third parties. One throat to choke, one place to get reliability right.
What the new pieces do
- Servicesproducer
- Any backend service that needs to tell a user something — an order shipped, someone liked your post, a login from a new device.
Step 1 · The front door
One API for everyone
If every service integrates Apple, Google, Twilio and an email provider on its own, you get four copies of the same fragile code — and no consistent way to throttle, template, or respect a user’s unsubscribe. How should services send?
A dozen services each need push, SMS and email. How do they integrate delivery?
Four copies of fragile provider code in every service, four places to fix every bug, and no consistent throttling, templating or unsubscribe. Provider quirks leak everywhere.
Better than nothing, but credentials, rate limits and retries still run in every caller’s process — you can’t enforce a global cap or audit centrally, and every service must redeploy to fix delivery.
Producers make one call and walk away. A single service owns channels, preferences, templates, retries and provider credentials — one place to get reliability and compliance right.
Put a single Notification API in front. A producer makes one call — POST /notify {user, type, data} — and walks away. The API owns everything downstream, so senders stay blissfully ignorant of how delivery actually happens.
What the new pieces do
- Notification APIbackend
- One front door every producer calls. It validates the request and hands it to the pipeline, so senders never talk to providers directly.
Step 2 · Don’t block the sender
Decouple with a queue
Calling Apple or Twilio can take hundreds of milliseconds and sometimes fails. If the API delivered inline, a slow provider would stall the producer’s request — and a provider outage would take the whole thing down. How should the API respond fast?
Calling Apple/Twilio takes ~100s of ms and sometimes fails. How does the API stay fast?
Now the producer’s request is hostage to the slowest provider, and a provider outage stalls or fails every caller. Synchronous delivery couples your latency to third parties you don’t control.
The API does a fast local write to a durable queue and returns; a worker fleet drains it at provider speed. Outages and slowness become a backlog that waits, not lost requests.
Unbounded threads vanish on a crash (losing in-flight sends), can’t apply backpressure, and leave no durable record. A queue gives durability, retries and flow control for free.
The API just validates and enqueues, then returns instantly. A pool of delivery workers drains the queue (Kafka) and does the slow work. Producer latency is now a fast local write; delivery happens asynchronously.
- <10msenqueue
- 1000s/singest
- asyncdelivery
What the new pieces do
- Delivery Workersworker
- Drain the queue, render the template, pick the channel and call the provider. A stateless fleet you scale to match send volume.
- Notification Queuebus
- Decouples the fast API from slow third-party delivery. The API drops a message and returns; workers drain the queue at their own pace.
Back of the envelope
- enqueue ≈ <10ms local write
- vs ~100s of ms per third-party provider call
- 1 API tier ⇒ 1000s of msgs/s
- workers scale out independently to match delivery rate
- outage ⇒ backlog in the log
- durable wait, zero loss — drained when capacity returns
Step 3 · Respect the human
Did they even want this?
Blasting every event to every channel is how you get users to mute you — or report you as spam, which wrecks your sender reputation. People want control: which categories, which channels, and not at 3am. How do you respect that?
How do you avoid spamming users into muting you or reporting spam?
Pushing OS-mute onto users tanks engagement and your sender reputation, and ignores quiet hours and legal opt-out requirements. Consent has to live in your system.
A global rule ignores that each user wants different things, and preferences change. Per-user, per-type, per-channel consent belongs in data, not code.
Every notification passes a consent gate that honors opt-outs, channel choice and quiet hours. It’s both good manners and the law (CAN-SPAM, GDPR).
Before sending, a worker checks the Preference Service: is this user opted in to this type, on this channel, right now? It enforces quiet hours and unsubscribes, dropping or deferring anything the user didn’t ask for.
What the new pieces do
- Preference Serviceservice
- Decides whether this user wants this kind of notification, on which channels, and whether it is their quiet hours. The unsubscribe gatekeeper.
Step 4 · Many shapes, many channels
Render once, fan out
“Order shipped” must become a 20-character push title, a 160-character SMS, and a full HTML email — in the user’s language. And it should go to every channel they’ve enabled, not just one.
Workers pull from a Template Store (variables + localization) to render the right copy per channel, then fan out to Push (APNs/FCM), SMS (Twilio) and Email (SES). Each channel is an adapter that speaks one provider’s protocol.
- 3+channels
- Nlanguages
- 1template
What the new pieces do
- Template Storestore
- Holds message templates with variables and localization, so a worker can render “Hi {name}, your order shipped” in any language.
- Pushchannel
- Delivers to phones via Apple (APNs) and Google (FCM). The most common channel and the most latency-sensitive.
- SMSchannel
- Short text to a phone number through a provider like Twilio. Expensive per message, so reserved for high-value alerts.
- Emailchannel
- Rich, cheap, and tolerant of delay. Handled by a transactional provider that manages reputation, bounces and spam folders.
Back of the envelope
- 1 event × channels enabled
- one notify fans out to push + SMS + email
- 1 template × N languages
- rendered per channel/locale from a single source
- new channel = new adapter
- Slack/WhatsApp added without touching any producer
Step 5 · Not all events are live
Scheduled & triggered sends
Plenty of notifications aren’t reactions to an event: a daily digest, a “your trial ends tomorrow” reminder, a re-engagement nudge after 7 days of silence. Nothing fires these unless something is watching the clock. How do you send them?
Digests and "trial ends tomorrow" reminders aren’t event reactions. How are they sent?
Time-based sends funnel through the identical pipeline — preferences, templates, channels, tracking — so a scheduled digest behaves exactly like a live alert. One path, uniform behavior.
Now consent, templating, tracking and rate limits exist twice and drift apart. A new source of notifications shouldn’t mean a new delivery path.
Scattering schedules across every service means no central view, duplicated cron logic, and inconsistent throttling. Centralize the clock the way you centralized delivery.
Add a Scheduler that emits notifications at the right time, calling the same Notification API as everything else. Reuse the entire pipeline — preferences, templates, channels — so scheduled sends behave identically to live ones.
What the new pieces do
- Schedulertrigger
- Fires time-based notifications — daily digests, reminders, drip campaigns — on a schedule rather than in reaction to a live event.
Step 6 · Did it actually arrive?
Track delivery & retry
Providers fail in a dozen ways: a device token expires, an email bounces, Twilio rate-limits you. Fire-and-forget means you never learn — and the user silently misses something important. How do you make delivery reliable?
Providers fail silently — expired tokens, bounces, rate limits. How do you not lose sends?
You never learn about failures, so users silently miss important alerts and you can’t debug "why didn’t I get it?". Delivery must be a measured fact, not a hope.
Permanent failures (dead token, hard bounce) never succeed — retrying them forever burns provider quota for nothing. You must distinguish transient from permanent.
Delivery Tracking makes every send auditable; transient failures retry with backoff (idempotently, so no dupes), and permanent ones are flagged to stop trying and fall back to another channel.
Every attempt writes its outcome to Delivery Tracking — sent, delivered, bounced, opened. Failures get retried with backoff; permanent ones (dead token, hard bounce) are marked so you stop trying and can fall back to another channel.
- 99.9%delivered
- ×3retry w/ backoff
- bouncefeedback loop
What the new pieces do
- Delivery Trackingstore
- Records every send’s outcome — delivered, bounced, opened — feeding retries, analytics, and the “why didn’t I get it?” debugging.
Back of the envelope
- transient fail ⇒ retry ×3 backoff
- rate-limited / token-expired recover on their own
- hard bounce ⇒ stop + fall back
- don’t burn provider quota on permanent failures
- every attempt ⇒ tracked outcome
- delivery becomes measurable, debuggable, auditable
Step 7 · The sharp edges
No duplicates, no floods
Retries, multiple producers, and at-least-once queues all conspire to send the same alert twice. And a buggy loop or a viral post can fan out thousands of notifications to one poor user in a minute. How do you tame both?
At-least-once queues can double-send, and a viral post can fire 500 alerts in a minute. Fix?
Users experience that as broken spam. At-least-once is unavoidable across third parties, but you can collapse it to effectively-once and cap floods — so you should.
A stable key collapses duplicate sends to one; per-user limits and rollups turn "🔥 liked your post" ×500 into "500 people liked your post." Effectively-once and flood-proof.
Exactly-once across external providers is a fantasy — the provider may deliver, then fail to ack. Aim for at-least-once + dedup on a stable key, which actually works.
Add a Dedup + Rate Limit guard. An idempotency key per logical notification collapses duplicates; per-user, per-type rate limits (and digest-rollups) cap the flood so “🔥 liked your post” ×500 becomes “500 people liked your post.”
What the new pieces do
- Dedup + Rate Limitguard
- Stops the same alert firing five times and caps how many notifications a user gets per hour, using idempotency keys.
You did it
You just designed a notification system.
Everything you assembled, in order
- One Notification API as the single front door for every producer.
- A queue + worker fleet so slow third-party delivery never blocks senders.
- A Preference Service enforcing opt-ins, channels and quiet hours.
- Template-driven rendering that fans out to push, SMS and email.
- A scheduler reusing the same pipeline for digests and reminders.
- Delivery tracking with retries and bounce feedback for reliability.
- Idempotency-key dedup and per-user rate limits to kill duplicates and floods.