The whole design, in writing
Learn system design by building a social news feed step by step. An interactive guide covering the social graph, pull vs push fan-out, a precomputed feed cache, ranking, the hybrid model for celebrities, mixing content sources, and pagination.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a news feed?
Open the app and you see a personalized stream of posts from everyone you follow, freshest and most relevant on top. Simple to use — but building one reader’s feed means merging posts from hundreds or thousands of authors, ranking them, in a few hundred milliseconds, for billions of readers.
The whole game is when you do the work: merge everyone’s posts at read time (simple, slow) or pre-build each feed at write time (fast reads, heavy writes). We’ll start naive and evolve into the hybrid that real feeds use.
What the new pieces do
- Authorclient
- Anyone who publishes. A normal user has hundreds of followers; a celebrity has millions — and that difference reshapes the whole design.
- Readerclient
- Pulls down to refresh and expects a fresh, ranked feed in a few hundred milliseconds. Reads vastly outnumber posts.
Step 1 · The skeleton
Post and read
Two operations: an author publishes a post, and a reader asks for their feed. Both go through one place, and the post content has to live somewhere durable. How should feeds refer to posts?
A post can appear in millions of feeds. How should those feeds store it?
Now an edit or delete must hunt down millions of copies, and storage explodes. Duplicated content turns every change into a distributed cleanup problem.
One durable copy in the Post Store; feeds hold only IDs and hydrate bodies at read time. Edits, deletes and privacy changes resolve correctly everywhere, for free.
That’s pure pull — every reader re-queries every author they follow on each refresh. Simple, but it makes the hot read path do all the work (the next step’s problem).
Stand up a Feed API with a Post Store behind it. Writing a post saves a row; reading a feed will (for now) go figure out what to show. This is the spine everything else hangs off.
What the new pieces do
- Feed APIbackend
- The entry point for both posting and reading. It assembles a reader’s feed and accepts an author’s new posts.
- Post Storestore
- The durable home of post bodies, media references and metadata. Feeds store only IDs; this is where the actual content is hydrated from.
Step 2 · The naive read
Pull (fan-out on read)
To build a feed on demand, you’d look up everyone the reader follows in the Social Graph, fetch each one’s recent posts, merge and sort them — every single time they refresh.
This pull model is dead simple and always fresh, and writes are trivial (just save the post). But a reader following 2,000 accounts triggers a 2,000-way query and merge on every open — far too slow at scale.
- 2,000follows to merge
- everyrefresh re-queries
- freshbut slow
What the new pieces do
- Social Graphservice
- Stores the follow edges. To build a feed you first need to know whose posts a reader should even see.
Back of the envelope
- follows 2,000 × recent posts
- a 2,000-way fetch + merge on every single refresh
- reads ≫ writes (~100:1)
- paying the big cost on the common operation is backwards
- writes are O(1)
- posting is just one insert — pull’s one redeeming virtue
Step 3 · Flip the work
Push (fan-out on write)
Reads are the hot path, yet pull does the most work there. We want reading a feed to be a single cheap lookup — which means the answer must already exist before the reader asks. When do you build the feed?
Reads dominate but pull does its heaviest work on reads. How do you make a feed read O(1)?
It helps a little, but the first reader still pays the full 2,000-way merge, and the cache is cold or stale exactly when new posts arrive. You’re patching pull, not fixing it.
Replicas scale raw query throughput but every read still does the expensive multi-author merge. You’ve made a slow operation parallel, not cheap.
When an author posts, push the ID into every follower’s feed cache. Reading is then O(1) — grab your precomputed list. Work moves from many reads to one write.
When an author posts, Fan-out Workers push the post ID into the Feed Cache of every follower. Reading is now O(1): grab your precomputed list of IDs and hydrate the bodies. Work moved from many reads to one write.
- O(1)feed read
- precomputedper follower
- ×followerswrite cost
What the new pieces do
- Feed Cachecache
- A ready-made list of post IDs per user. Reading the feed becomes a single fast lookup instead of a live query across everyone they follow.
- Fan-out Workersworker
- When someone posts, these workers push the post ID into each follower’s feed cache — doing the join once at write time, not on every read.
Back of the envelope
- 1 post × F followers
- F cache inserts at write time — the amplification
- typical F ≈ hundreds
- cheap: a few hundred tiny ID inserts per post
- celebrity F ≈ 50M
- 50M inserts for one post — push collapses here (step 5)
Step 4 · Newest isn’t best
Ranking
A purely chronological feed buries the post you’d most want to see under noise. Engagement craters when relevance is left to luck and timestamps. How do you order the feed?
A chronological feed buries the best post. How do you order what the reader sees?
Simple and predictable, but a single chatty account drowns the post you actually care about. Recency alone is a weak proxy for relevance.
That hands ranking to the people with the most incentive to game it — every post becomes "top priority". Ordering must be decided by the platform, from signals, not by the author.
Two stages: cheaply gather candidates, then a model orders just those by how close you are to the author, freshness and likely engagement. The same retrieve-then-rank pattern powers search.
Run candidates through a Ranking stage that scores each by affinity (how close you are to the author), recency, and predicted engagement. The feed cache holds candidates; ranking decides their order at (or near) read time.
What the new pieces do
- Rankingservice
- Scores candidate posts by relevance — affinity, recency, engagement — so the feed shows what matters, not just the newest thing.
Step 5 · The celebrity problem
Go hybrid
Push breaks for the mega-popular: one post by a celebrity with 50M followers means 50M cache writes — a thundering, slow, expensive fan-out that also wastes effort on inactive followers. What do you do for them?
A celebrity with 50M followers posts. Push to all 50M feeds, or something else?
50M cache writes per post, per celebrity, including for followers who’ll never open the app. You can parallelize it, but it’s enormous wasted work on the write path.
Now every ordinary reader pays the slow multi-author merge again — you’ve thrown away the O(1) reads that push bought you, to fix a problem only celebrities have.
Skip fan-out for the few mega-accounts and merge their recent posts into the precomputed feed on read. The merge set is tiny (a handful of celebs) so reads stay fast and writes stay sane.
Go hybrid. Push for ordinary authors; for celebrities, skip fan-out and let readers pull their recent posts at read time, merging them into the precomputed feed. Route new posts through an event stream so fan-out can absorb spikes and run async.
- pushnormal authors
- pullcelebrities
- mergeat read time
What the new pieces do
- Post Eventsbus
- A stream of new posts. Fan-out, search indexing and analytics all consume it independently — and it absorbs celebrity-sized spikes.
Back of the envelope
- a 10k-follower author ⇒ 10k inbox writes per post
- expensive, but bounded — push is still the right call
- a 50M-follower account ⇒ 50M writes for that same single post
- 5,000× the cost of the author above — this one number is why the cutoff exists
- so push below ~10k followers, pull above it
- the threshold is not a convention; it is where write cost stops being survivable
- pull the few thousand mega-accounts
- their posts are fetched and merged on read
- read = precomputed feed + N celeb pulls
- N is single digits, so reads stay fast
Step 6 · More than friends
Mixing sources
A modern feed isn’t only people you follow — it’s followed pages, recommended posts you don’t follow yet, and ads. These come from different systems but must feel like one coherent stream. How do you combine them?
Friends, pages, recommendations and ads all need to share one feed. How?
That’s not a feed, it’s a dashboard — and it ignores relevance across sources. A great recommended post should be able to outrank a dull friend post, which fixed sections forbid.
Friends, pages, recs and ads each emit candidates; a shared ranker scores and interleaves them under one policy (with ad-spacing and diversity rules). Adding a new source is a new generator, not a new feed.
Phones can’t see global ranking signals, the merge logic forks across platforms, and you ship ranking secrets to the client. Blending belongs on the server, in one place.
Gather candidates from each source — friends (feed cache), pages, recommendations, ads — then rank and interleave them under one policy, with rules for ad spacing and diversity so the feed doesn’t clump.
What the new pieces do
- Other Sourcesstore
- Followed pages, recommended posts and ads. The final feed blends these with friends’ posts under one ranking.
Step 7 · The sharp edges
Pagination, dedup & freshness
Readers scroll for ages, so feeds must page without showing the same post twice or skipping new ones that arrive mid-scroll. And a post deleted after fan-out shouldn’t haunt a million feeds. How do you page an ever-changing feed?
Infinite scroll, with posts constantly inserted and deleted. How do you paginate?
When new posts arrive at the top, every offset shifts — page 2 now repeats items from page 1, or skips some. Offset pagination and a live feed don’t mix.
A stable cursor marks your position so inserts can’t cause dupes or gaps; dedup drops already-seen IDs; and because feeds store IDs, hydration re-checks the Post Store so deleted/edited posts resolve correctly at read time.
Refetching everything is wasteful, janky, and still doesn’t give a stable scroll position — the reader keeps losing their place as the top churns.
Use cursor-based pagination (a stable position, not page numbers) so inserts don’t cause dupes or gaps. De-dupe already-seen IDs per session. Because feeds store IDs, hydration re-checks the Post Store — so deleted or edited posts resolve correctly at read time.
You did it
You just designed a news feed.
Everything you assembled, in order
- A Feed API over a Post Store; feeds reference posts by ID, never copy bodies.
- Pull (fan-out on read) is simple and fresh but slow for big follow counts.
- Push (fan-out on write) precomputes feeds for O(1) reads at write-time cost.
- A ranking stage orders candidates by affinity, recency and engagement.
- A hybrid model pushes for normal authors and pulls celebrities to dodge mega-fan-out.
- Multiple candidate sources (pages, recs, ads) blended under one ranker.
- Cursor pagination, per-session dedup and ID hydration keep an infinite scroll correct.