The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is an email system, systems-wise?
Email is unusual among the systems in this series: it's federated. Your mail server has to talk to millions of OTHER organizations' mail servers you don't control, over a protocol (SMTP) with no guaranteed delivery, no built-in spam prevention, and no shared trust — while still giving your own users a fast, reliable, always-in-sync inbox.
We'll build both sides: the inbound path (accept mail from an untrusted internet, filter it, store it) and the outbound path (send reliably to servers that may reject, delay, or vanish), plus what makes an inbox actually usable at scale — sharded storage, search, threading, and sync across devices.
Step 1 · The baseline
Accept mail, store it
Simplest version: accept any incoming SMTP connection, take the message, and write it straight into the recipient's mailbox. What's obviously missing?
Mail arrives and is written directly into the mailbox with no intermediate checks. What's the glaring gap?
SMTP as a protocol has essentially no built-in sender verification — anyone can claim to be sending "from" any address unless additional authentication mechanisms are checked, which this baseline doesn't do.
Without any filtering between "accept the SMTP connection" and "it's in the inbox," every inbox is wide open to spam, phishing, and spoofing at internet scale. This is the single most important thing missing from the naive version.
Storage format efficiency is a real but secondary concern — it doesn't compare to the fundamental gap of having zero defenses against malicious or unwanted mail.
The glaring gap is trust: SMTP alone provides no sender verification and no content screening, so a direct accept-and-store pipeline puts every inbox one connection away from spam, phishing, and spoofed senders. Everything received from the open internet needs to pass through filtering before it's considered "delivered."
Why this piece earns its place
Notice what the front door commits you to. The moment the MTA answers 250 OK it has taken responsibility for the message — the sending server drops it from its queue and stops caring. If you discover afterwards that the local part doesn’t exist, the mailbox is full, or the recipient left the company, your only way to tell anyone is to generate a bounce, and the address you would send it to was never verified. On the open internet that address is routinely forged, so accept-then-bounce turns your server into a machine that mails strangers about spam they never sent — and the receiving world scores that against your IP, not the forger’s. The defense is to make every rejection you can happen while the sending server is still on the line: validate the recipient at RCPT TO and refuse there, permanently. Then the rejection is the sender’s problem to report, it costs you no outbound mail, and the only bounces you ever generate are for messages you genuinely accepted and genuinely could not deliver.
What the new pieces do
- External Mail Systemclient
- Any other mail server on the internet — sending to your users, or receiving what your users send out. Federated, and not under your control.
- Inbound MTAgateway
- Accepts incoming mail over SMTP from any mail server on the internet — the front door for everything not yet trusted.
Step 2 · Sending is unreliable — plan for it
Queue and retry, don't send-and-forget
A user sends an email. The recipient's mail server is temporarily down, or slow, or greylisting the message (a common anti-spam tactic: reject once, accept on retry). Should the send simply fail if the first attempt doesn't succeed?
The first delivery attempt to a recipient's mail server fails or times out. What should happen?
A single failed attempt — especially to an external server outside your control — is extremely common and often transient (temporary unavailability, greylisting, rate limiting). Failing immediately on the first attempt would make sending mail feel unreliable for a problem that's usually self-resolving.
Because delivery across the open internet to servers you don't control is inherently unreliable, a durable queue with backoff-retry absorbs the common transient failures automatically. Only after real, sustained failure (or an explicit permanent rejection like "mailbox doesn't exist") does the system give up and notify the original sender via a bounce.
Retrying with no backoff can itself look like abuse to the receiving server (and waste resources hammering a server that's genuinely down) — backoff, not immediate relentless retry, is the correct pattern for a failure that may take time to resolve.
Queue outbound mail durably and retry with exponential backoff over a bounded window — this absorbs the common, transient delivery failures (temporary unavailability, greylisting) automatically. Only once delivery is genuinely, permanently impossible (or the retry window expires) does the sender get notified, via a bounce message.
Why this piece earns its place
The retry schedule is the knob, and its two ends fail differently. The first retries should sit close together, because the most common reason a first attempt fails is greylisting — the receiving server rejects a stranger once on purpose and accepts the same message minutes later. Back off hard from the start and you turn a deliberate short delay into hours. Widen the gaps after that, and bound the window: a queue that retries forever is a warehouse of mail nobody will read, while the user still believes it was sent. Give up too early and you bounce for an outage that would have cleared on its own, which is worse, because a bounce is final from the user’s side — they have to decide to send it again. Two things make the window livable: a delay notification partway through, so the sender knows it is still trying rather than silently stuck, and a separate queue per destination domain, so one unreachable provider’s backlog cannot starve delivery to everyone else.
- durable queuenot fire-and-forget
- exponential backoffnot immediate hammering
- bounceonly after real, confirmed failure
What the new pieces do
- Our Userclient
- A user of this mail system, reading their inbox from possibly several devices at once and expecting all of them to stay in sync.
- Outbound MTAgateway
- Queues outgoing mail and retries delivery with backoff — because delivering to an arbitrary external mail server is inherently unreliable.
Step 3 · Layered filtering
Cheap sender checks before expensive content analysis
Every accepted message needs to be screened for spam and phishing before it reaches a mailbox. Should every message run through the same, presumably expensive, ML content-classification model?
Screening every inbound message for spam/phishing — run the expensive ML model on all of it?
Running an expensive content-analysis model on 100% of inbound traffic, including obviously-spoofed mail that fails basic sender authentication, wastes enormous compute on messages that a much cheaper check would have already caught.
SPF/DKIM/DMARC verify sender authenticity essentially for free and catch a large fraction of spoofed and bulk-spam traffic instantly. Only the messages that pass basic authentication — genuinely ambiguous cases where content actually needs to be judged — reach the costly ML stage, keeping overall throughput far higher.
Manual review can't remotely keep pace with internet-scale inbound mail volume — automated, layered filtering has to do the vast majority of the work, with any human-in-the-loop process reserved for edge cases or appeals.
Layer the filter: SPF/DKIM/DMARC sender-authentication checks run first — cheap, deterministic, and effective at rejecting a large share of spoofed/bulk traffic instantly. Only messages that pass those checks proceed to ML-based content analysis, which is reserved for the genuinely ambiguous cases that actually need judgment.
Why this piece earns its place
The layering rests on an assumption worth saying out loud: that bad mail is cheap to spot. That holds for the crude flood — forged envelopes, senders who never published a record — which is exactly why the cheap tier absorbs a bulk campaign. It stops holding the day the attacker does the paperwork. Registering a fresh domain and publishing valid SPF and DKIM records is nearly free, and a compromised account at a real provider passes every sender check by construction, because the mail genuinely is from where it claims. Authenticated spam sails through the cheap tier and lands on the expensive one, so the layer that was shedding load stops shedding. What makes authentication still pay is that it converts an unverifiable claim into a stable identity you can keep a score against: reputation per authenticated domain and per sending IP, aged and earned, so a brand-new domain sending a surge is throttled at the connection before its content is ever read. And keep the cheap tier’s rejections deterministic — reject on a published policy, never on a hunch, because a wrongly rejected legitimate message is the one loss a user never forgives.
What the new pieces do
- Spam / Auth Filterservice
- Cheap sender-authentication checks (SPF/DKIM/DMARC) first, expensive ML content analysis only for what survives them.
Step 4 · Shard the mailbox by user
Every query is "my mailbox," never a cross-user join
Billions of users, each with thousands of messages. How should the mailbox store be partitioned so that reading "my inbox" stays fast at any scale?
How should mailbox storage be partitioned for a system with billions of users?
Email access is almost perfectly partition-friendly: a user only ever reads their own mailbox. Sharding by user id means any single user's query touches exactly one (or a small, known set of) partition, and the system scales by adding more partitions as the user base grows — no query ever needs to fan out across the whole dataset.
Date-based sharding would mean a single user's inbox is scattered across many partitions (their messages arrived on many different dates), so reading "my inbox" would require querying nearly every partition — the opposite of what you want.
A single unsharded table for billions of users' worth of messages doesn't scale in either storage or query throughput — some partitioning strategy is required, and user id is the natural key given the access pattern.
Shard the Mailbox Store by user id. Because virtually every real access pattern is "show me my own mail" — never a query spanning multiple users' mailboxes — user-id sharding means any single query touches only the partition(s) holding that one user's data, and the system scales horizontally simply by adding more partitions as the user base grows.
Why this piece earns its place
The shard key answers the read path; the write path is where it bites. One message to a large distribution list is a single inbound SMTP transaction that becomes a write into every recipient’s partition, and those partitions fail independently. Hold the sending server on the line until all of them commit and you are as slow and as available as your worst shard — and it will retry a message you half-delivered. So separate the moment of acceptance from the moment of delivery: make the message durable once, acknowledge, then fan out per recipient asynchronously, each with its own retry state and its own final status. That changes what “delivered” means — it is per recipient, not per message, so the bounce path has to be able to say one address failed while the rest succeeded. The corollary for reads is that any cross-user question you might later want — an admin view of one campaign, abuse investigation across mailboxes — has to come from a separate index, because the store itself was deliberately arranged so no query ever spans users.
What the new pieces do
- Mailbox Storedata
- Durable per-user storage — sharded by user id, because every real access pattern is "my mailbox," never a cross-user query.
Step 5 · Search and threading
Group a conversation without trusting the subject line
A user wants to see an entire email conversation as one thread. Subject lines can collide ("Re: Meeting" exists in a thousand different, unrelated conversations) or get manually edited. How do you reliably group messages into the SAME conversation?
How do you reliably group messages into one conversation thread, not just by matching subject text?
Subject lines are ambiguous (many unrelated conversations share a common subject like "Re: Meeting") and mutable (someone can edit the subject mid-thread), making them an unreliable grouping key on their own.
The same two people can have many completely separate conversations over time — sender/recipient alone doesn't distinguish "which" conversation a given message belongs to.
Every properly-formed reply email includes headers that explicitly reference the Message-ID of the message(s) it's replying to — this is a direct, protocol-level causal link, not a text-matching heuristic. Following these references reconstructs the actual conversation tree correctly even when subjects collide or get edited.
Reconstruct threads from the References and In-Reply-To headers — each reply explicitly links to the Message-ID(s) it's responding to, forming a reliable causal chain. Combine this with a per-user search index (full-text over subject/body, scoped to that user's shard) so both search and threading stay fast and accurate at scale.
Why this piece earns its place
Header threading is right, and it still has to survive clients that get it wrong. Chains break from underneath: a mailer that drops References on reply, a user who starts a fresh message instead of replying, a forwarded copy carrying no link back — each strands a message outside the conversation it belongs to. Chains also merge wrongly, because those headers are sender-supplied: a client that reuses a Message-ID, or a reply carrying a long inherited reference list, can stitch two unrelated conversations into one. The protection is to keep the parent link on the message and compute the thread when you read or index it, rather than stamping a thread id on at delivery. A derived thread can be recomputed the day you improve the rule; a stored one is a mistake you have to migrate out of. Where the headers are genuinely absent, any fallback stays narrow — same participants, close in time, never subject text on its own — and the thread is assembled per user inside that user’s shard, because two people in one conversation hold different subsets of it and neither view is the global truth.
What the new pieces do
- Search + Threadingdata
- Full-text search plus conversation threading, reconstructed from message headers rather than a possibly-ambiguous subject line.
Step 6 · Never lose a message, never bounce forever
Delivery dedup and the bounce loop
A retry after a network blip might have actually succeeded the first time — risking a duplicate delivery. And a permanently-failing message generates a bounce back to the sender — but what if the bounce ITSELF can't be delivered?
Attach a unique Message-ID to every message and use it for delivery deduplication — the recipient system recognizes and discards a true duplicate rather than delivering it twice. For bounces specifically: a bounce message is marked with a special empty return-path (so it can never itself generate another bounce), which structurally prevents an undeliverable-bounce from looping forever between two mail servers.
Why this piece earns its place
Dedup by Message-ID is only as trustworthy as the id, and the id is written by whoever sent the mail. Two genuinely different messages can arrive carrying the same one — from a broken client, or from someone who noticed that a repeated id makes the second message disappear. Silently dropping mail is the failure this system is least allowed to have, so the real key is the id together with the envelope sender and recipient, and a content hash is what settles the case where those match but the bodies do not. Bound the memory, too: you only need to recognize a duplicate within the window that could produce one, and a table that remembers forever both grows without end and makes a false match likelier the longer it runs. When it is genuinely ambiguous, deliver twice — a user can see two copies and understand what happened, whereas a message that never arrives looks exactly like a message that was never sent. The same asymmetry sits on the bounce path: an empty return-path ends the recursion, but it says nothing about volume, so bounce generation still needs its own rate limit.
Step 7 · Multi-device sync
The same inbox, several devices, always agreeing
A user reads an email on their phone. It should show as read on their laptop too, without the laptop re-downloading the entire mailbox to notice.
Use an incremental sync protocol (the same principle IMAP uses): each device tracks a position/token representing what it has already seen, and asks for "what changed since then" — new messages, flag changes (read/unread, starred), folder moves. A device that reconnects after being offline for days still just requests one delta, not a full mailbox re-fetch.
Why this piece earns its place
A cursor only means something while the server still holds the history it points into. The change log is not kept forever — it gets compacted, trimmed, or rebuilt with a shard — and a device that has been off for a long trip comes back holding a token that no longer resolves. If the server quietly answers “nothing changed since then,” that device silently misses the whole gap and stays wrong indefinitely, which is worse than being obviously stale. So the protocol needs an explicit answer for it: this cursor is too old, resynchronize from scratch. Retention then becomes a real cost decision — long enough that a full re-fetch is rare rather than routine, short enough to pay for. Two details keep the deltas honest. Deletions have to be recorded as tombstones, because a delta can only report rows that exist, and a message removed without a trace is invisible to every device that did not watch it go. And conflicts need a rule per field rather than one blanket rule: last write wins is fine for read and unread, but two devices moving the same message to different folders is a decision, and losing one of them quietly is how mail goes missing without anyone deleting it.
What the new pieces do
- Sync Serviceservice
- Keeps every device's view of read/unread state and folders consistent via incremental sync, not a full mailbox re-fetch.
Step 8 · The sharp edges
Attachments, floods, and bounce loops
Three more real-world pressures: large binary attachments shouldn't live inline in structured mailbox storage, spam campaigns can flood the system, and the bounce-loop risk from Step 6 needs to actually hold under adversarial conditions.
Store attachments in object storage (not inline in the mailbox database), with the mailbox record holding just a reference — the same large-binary-blob pattern used everywhere structured storage and large files meet. Under a spam/phishing flood, the layered filter from Step 3 does the heavy lifting: cheap sender-authentication checks reject the bulk of it before the expensive ML stage is ever touched, keeping filter throughput high under attack. And the bounce-loop prevention from Step 6 — a bounce that structurally can't generate another bounce — holds regardless of how adversarial or malformed an incoming message is.
Why this piece earns its place
Splitting a message across two stores gives every write and every delete an order that matters. Write the blob to object storage first and the mailbox record that references it second: a crash between the two leaves an unreferenced object, which a background sweep can reclaim, while the opposite order leaves a message the user can see in their list and cannot open. Deletion runs the other way and is not really a delete — one object backs every recipient’s copy of the same message as well as the sender’s, so removing a mailbox row can only drop a reference, and the object goes when the last reference does. Put that reclamation in a sweep rather than in the user’s delete path, so nobody waits on it and a lost race costs storage instead of data. The last piece is access: the object store knows nothing about mailboxes, so the only thing between a link and the whole corpus is the URL itself. Issue those short-lived and scoped to the user whose mailbox the reference was read from, never as a stable address that can be shared or guessed.
You did it
You just designed an email system.
Everything you assembled, in order
- Inbound mail from the open internet is untrusted by default and must pass filtering before it's considered delivered.
- Outbound delivery is inherently unreliable — a durable queue with exponential backoff absorbs transient failures automatically, with a bounce only after real, confirmed permanent failure.
- Spam/auth filtering is layered: cheap SPF/DKIM/DMARC sender checks first, expensive ML content analysis reserved for what survives them.
- Mailbox storage is sharded by user id — the access pattern is always "my mailbox," never a cross-user query.
- Threads are reconstructed from References/In-Reply-To headers, a reliable structural signal, not fragile subject-line text matching.
- Message-ID-based dedup prevents duplicate delivery from retries; a bounce is structurally prevented from ever generating another bounce.
- Multi-device sync uses an incremental token/cursor into a change log, never a full mailbox re-fetch.
- Attachments live in object storage with a reference from the mailbox record, not inline in structured storage.