The big idea

What is a notification system?

Across a product, dozens of services have something to tell a user: your order shipped, someone messaged you, your password changed. Each one should not have to know about Apple’s push servers, Twilio, or your email provider.

Build one service every team calls with “notify user U about event E.” It figures out the who, what and how — the right channels, the user’s preferences, the templates — and handles the messy delivery to third parties. One throat to choke, one place to get reliability right.

How to read this: We add one piece at a time. Each step opens with a problem, then the smallest thing that solves it. Watch the diagram grow — by the end you’ll have built the whole pipeline. Hit Begin.

Step 1 · The front door

One API for everyone

If every service integrates Apple, Google, Twilio and an email provider on its own, you get four copies of the same fragile code — and no consistent way to throttle, template, or respect a user’s unsubscribe. How should services send?

Design decision: A dozen services each need push, SMS and email. How do they integrate delivery?

The call: One Notification API every service calls with (user, type, data). — Producers make one call and walk away. A single service owns channels, preferences, templates, retries and provider credentials — one place to get reliability and compliance right.

Put a single Notification API in front. A producer makes one call — POST /notify {user, type, data} — and walks away. The API owns everything downstream, so senders stay blissfully ignorant of how delivery actually happens.

A single entry point: Centralizing behind one service means one place for retries, rate limits, auditing and provider credentials. Producers depend on a stable internal contract, not on the quirks of APNs.

Step 2 · Don’t block the sender

Decouple with a queue

Calling Apple or Twilio can take hundreds of milliseconds and sometimes fails. If the API delivered inline, a slow provider would stall the producer’s request — and a provider outage would take the whole thing down. How should the API respond fast?

Design decision: Calling Apple/Twilio takes ~100s of ms and sometimes fails. How does the API stay fast?

The call: Validate, enqueue, return instantly; workers deliver async. — The API does a fast local write to a durable queue and returns; a worker fleet drains it at provider speed. Outages and slowness become a backlog that waits, not lost requests.

The API just validates and enqueues, then returns instantly. A pool of delivery workers drains the queue (Kafka) and does the slow work. Producer latency is now a fast local write; delivery happens asynchronously.

Producer/consumer: A queue lets a fast producer and slow consumers run at different speeds. If providers lag or workers crash, messages wait durably in the log and get processed when capacity returns — nothing is lost.

Step 3 · Respect the human

Did they even want this?

Blasting every event to every channel is how you get users to mute you — or report you as spam, which wrecks your sender reputation. People want control: which categories, which channels, and not at 3am. How do you respect that?

Design decision: How do you avoid spamming users into muting you or reporting spam?

The call: Check a Preference Service before every send (opt-in, channel, quiet hours). — Every notification passes a consent gate that honors opt-outs, channel choice and quiet hours. It’s both good manners and the law (CAN-SPAM, GDPR).

Before sending, a worker checks the Preference Service: is this user opted in to this type, on this channel, right now? It enforces quiet hours and unsubscribes, dropping or deferring anything the user didn’t ask for.

Preferences as a gate: Every notification passes a consent check. This isn’t just polite — channel reputation and, increasingly, the law (CAN-SPAM, GDPR) require honoring opt-outs. Default to the user’s wishes, not the product’s.

Step 4 · Many shapes, many channels

Render once, fan out

“Order shipped” must become a 20-character push title, a 160-character SMS, and a full HTML email — in the user’s language. And it should go to every channel they’ve enabled, not just one.

Workers pull from a Template Store (variables + localization) to render the right copy per channel, then fan out to Push (APNs/FCM), SMS (Twilio) and Email (SES). Each channel is an adapter that speaks one provider’s protocol.

Channel adapters: Hiding each provider behind a uniform adapter means adding WhatsApp or Slack later is a new adapter, not a rewrite. Templates keep copy consistent and translatable across all of them.

Step 5 · Not all events are live

Scheduled & triggered sends

Plenty of notifications aren’t reactions to an event: a daily digest, a “your trial ends tomorrow” reminder, a re-engagement nudge after 7 days of silence. Nothing fires these unless something is watching the clock. How do you send them?

Design decision: Digests and "trial ends tomorrow" reminders aren’t event reactions. How are they sent?

The call: A Scheduler that calls the same Notification API on a cron. — Time-based sends funnel through the identical pipeline — preferences, templates, channels, tracking — so a scheduled digest behaves exactly like a live alert. One path, uniform behavior.

Add a Scheduler that emits notifications at the right time, calling the same Notification API as everything else. Reuse the entire pipeline — preferences, templates, channels — so scheduled sends behave identically to live ones.

Reuse the pipeline: A new source of notifications shouldn’t mean a new path. Funnel cron jobs and event triggers through the same API so consent, templating and tracking apply uniformly.

Step 6 · Did it actually arrive?

Track delivery & retry

Providers fail in a dozen ways: a device token expires, an email bounces, Twilio rate-limits you. Fire-and-forget means you never learn — and the user silently misses something important. How do you make delivery reliable?

Design decision: Providers fail silently — expired tokens, bounces, rate limits. How do you not lose sends?

The call: Record each outcome, retry transient failures with backoff, mark permanent ones. — Delivery Tracking makes every send auditable; transient failures retry with backoff (idempotently, so no dupes), and permanent ones are flagged to stop trying and fall back to another channel.

Every attempt writes its outcome to Delivery Tracking — sent, delivered, bounced, opened. Failures get retried with backoff; permanent ones (dead token, hard bounce) are marked so you stop trying and can fall back to another channel.

At-least-once + status: Combine retries with idempotency so a retried message isn’t a duplicate. Tracking turns delivery from a hope into a measurable, debuggable, and auditable fact.

Step 7 · The sharp edges

No duplicates, no floods

Retries, multiple producers, and at-least-once queues all conspire to send the same alert twice. And a buggy loop or a viral post can fan out thousands of notifications to one poor user in a minute. How do you tame both?

Design decision: At-least-once queues can double-send, and a viral post can fire 500 alerts in a minute. Fix?

The call: Idempotency-key dedup + per-user/type rate limits with digest rollups. — A stable key collapses duplicate sends to one; per-user limits and rollups turn "🔥 liked your post" ×500 into "500 people liked your post." Effectively-once and flood-proof.

Add a Dedup + Rate Limit guard. An idempotency key per logical notification collapses duplicates; per-user, per-type rate limits (and digest-rollups) cap the flood so “🔥 liked your post” ×500 becomes “500 people liked your post.”

Design for the unhappy path: Exactly-once is a fantasy across third parties — so aim for effectively-once: at-least-once delivery plus dedup on a stable key. Rate limits protect both the user and your provider quotas.

You did it

You just designed a notification system.

• O — n
• A —
• A —
• T — e
• A —
• D — e
• I — d