The whole design, in writing
Learn AI system design by building a large-scale recommendation system step by step. An interactive guide covering the two-stage retrieve-and-rank funnel, two-tower embedding retrieval with ANN, a heavy ranking model, the feature store and its offline/online split, filtering business rules, and the feedback loop that keeps recommendations fresh.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Pick the best few from millions
A user opens the app. Somewhere in a catalog of millions of items are the ten they’d most want to see right now. You have tens of milliseconds to find them. You can’t score millions of items per request — but you can’t guess, either. How do you do this fast and well?
Use a funnel. A cheap, fast stage narrows millions to a few hundred plausible candidates; a heavy, accurate stage ranks those few hundred precisely. Cheap-and-wide, then expensive-and-narrow — the pattern behind every large recommender.
What the new pieces do
- Userclient
- A person who opens the app expecting a feed of items they’ll actually like — picked from millions.
Step 1 · The skeleton
A request and a funnel
The user opens the app and needs a feed. We can’t run a heavy model over millions of items in the time budget. What overall structure makes "best ten from millions" feasible?
You must return the best 10 of millions in ~tens of ms. What’s the shape?
Running a heavy model over millions of items per request is impossible in the latency budget. You must shrink the set first.
Popularity is a fine fallback but ignores the individual — it’s not personalization, and engagement suffers. You need per-user relevance.
Stage 1 narrows millions to hundreds cheaply; stage 2 ranks those hundreds with a heavy model. Each stage is sized for its job.
A Recs Gateway runs a two-stage funnel for every request: candidate generation (millions → hundreds, fast) then ranking (hundreds → ordered, accurate), with filtering before the final list ships to the user. The shape is the whole idea.
What the new pieces do
- Recs Gatewaybackend
- Runs the funnel for each request: fetch candidates, rank them, apply rules, return the final list.
Step 2 · Represent taste
Embeddings for users and items
To find items a user will like, you need to compare a user against items numerically. "Action movies" and "this thriller" need to be close somehow. How do you represent taste so similarity is computable?
How do you make "this user" and "this item" comparable for similarity?
Tags are coarse and miss latent taste — two thrillers can feel totally different. Learned embeddings capture nuance hand-tags can’t.
Train embeddings from engagement so a user vector lands near the item vectors they’d enjoy. Similarity becomes distance in that space.
IDs carry no notion of similarity — item 5012 isn’t "near" item 5013. You need a learned space where proximity means relevance.
The system learns embeddings: users and items map to vectors in a shared space where a user sits near the items they’d engage with. These come from training on past engagement, so proximity encodes learned taste — not hand-typed tags.
- millionsitems, each a vector
- 1 vectorper user
- distance= predicted affinity
Step 3 · Narrow the field
Two-tower retrieval with ANN
You have a user vector and millions of item vectors. You need the few hundred closest — in milliseconds, per request. Comparing against every item exactly is too slow. How do you retrieve candidates fast?
Find the few hundred closest items to the user vector, fast. How?
Exact scoring over millions per request blows the latency budget. At this scale you use an approximate index.
A "two-tower" model produces the user vector live and item vectors offline; an ANN index returns the nearest few hundred in milliseconds.
Category filters are a blunt pre-filter that miss cross-category gems and still leave too many to score exactly. ANN over embeddings is the scalable answer.
A two-tower model has a user tower (computed live from the request) and an item tower (item vectors precomputed offline and stored in the Item Index). Candidate Generation embeds the user, then runs ANN search to pull the nearest few hundred items in milliseconds. Recall stage: don’t miss the good ones.
What the new pieces do
- Candidate Genservice
- Narrows millions of items to a few hundred plausible candidates fast, using embedding similarity.
- Item Indexindex
- Vector index of every item’s embedding, so candidate generation is a nearest-neighbour lookup.
Step 4 · Rank with care
The heavy ranking model
Retrieval handed back ~hundreds of candidates, ordered only by rough embedding similarity. Similarity isn’t the same as "will this specific user engage right now." How do you get the order right?
You have ~hundreds of candidates. How do you order them precisely?
Embedding distance is a coarse recall signal, not a precise engagement prediction. The top few hundred need a real scoring model.
Popularity ignores the individual and context. The whole point of ranking is per-user, per-moment precision.
A ranking model scores P(engage) for each candidate using many user/item/context features — affordable now because there are only hundreds, not millions.
A heavy Ranking Model scores each candidate’s probability of engagement (click, watch, purchase) using rich features — user history, item attributes, context (time, device), cross features. It’s far too expensive to run over millions, but perfect over a few hundred. Precision stage: order them right.
What the new pieces do
- Ranking Modelservice
- A heavy model that scores each candidate’s probability of engagement using rich user/item features.
Step 5 · Feed the model
The feature store
Ranking needs features — "how many cooking videos did this user watch today?", "this item’s 1-hour click-rate". Some are slow to compute; all must be fresh and identical between training and serving. Where do features come from?
Ranking needs fresh features, and training must see the SAME values. How?
Heavy aggregations can’t be computed within the latency budget per request. And ad-hoc computation drifts from what training saw.
Compute features in an offline pipeline AND serve them online with low latency, from one definition — so training and serving see identical values (no skew).
Features are dynamic (today’s watch count); they can’t be frozen into weights. They must be served live and refreshed.
A Feature Store serves ranking its features and has two halves that must agree: an offline pipeline computes features in batch (for training), and an online store serves the freshest values at request time (for serving) — from a single definition, so both see the same numbers.
What the new pieces do
- Feature Storestore
- Serves the user/item/context features ranking needs, with offline (batch) and online (low-latency) halves.
Back of the envelope
- offline half
- batch-compute features → training data
- online half
- serve freshest values at request, low latency
- one definition
- same logic both sides → no train/serve skew
- freshness matters
- today’s behavior must reach ranking today
Step 6 · The last mile
Filters, rules and diversity
Ranking gives a perfectly-ordered list — but the top items might be things the user already saw, items that violate policy, or ten near-identical videos. A great score isn’t a shippable feed. What sits between ranking and the screen?
The ranked list is perfect by score. Why isn’t it ready to ship?
Top-by-score can repeat already-seen items, surface blocked content, and stack near-duplicates. Real feeds need post-ranking rules.
Diversity matters, but blindly maximizing it tanks relevance. It’s one constraint among several applied after ranking, not a replacement for it.
A filtering stage removes already-seen/blocked items, enforces policy and freshness, and spreads diversity — turning a scored list into a shippable feed.
A Filters & Rules stage takes the ranked list and makes it shippable: drop already-seen and blocked/policy-violating items, enforce diversity (don’t show ten near-identical items), and apply business rules (freshness, sponsored slots, fairness). Then the final feed goes to the user.
What the new pieces do
- Filters & Rulesservice
- Removes already-seen, blocked, or policy-violating items and applies diversity/business constraints.
Step 7 · Close the loop
Log engagement, retrain, repeat
The feed shipped. The user clicks some things, ignores others. That reaction is the single most valuable signal you have — it’s both the truth about what’s good and the fuel for tomorrow’s models. How do you use it?
The user reacts to the feed. What do you do with that signal?
Ignoring engagement freezes the system — it never learns the user’s shifting taste. The feedback loop is what keeps recs alive.
Engagement events become training data and freshness signals: retrain the models periodically and refresh features so the system keeps improving.
Without impressions you can’t tell "shown and ignored" from "never shown" — you lose the negatives the model needs to learn from.
Every impression and interaction is logged to the Engagement Log. A Training Pipeline periodically retrains the retrieval and ranking models on that data and refreshes the feature store — so the system learns from what worked and adapts as taste shifts. The loop is the product.
What the new pieces do
- Engagement Logbus
- Every impression and interaction, logged — the training data and freshness signal for the whole system.
- Training Pipelinebus
- Periodically retrains the retrieval and ranking models on logged engagement so they keep improving.
The payoff
You built a recommender
From "best ten of millions" to a self-improving funnel: two-tower retrieval, a heavy ranker, a consistent feature store, post-ranking rules, and a feedback loop that retrains on engagement.
Now stale the feature store and watch relevance quietly rot — recommendations built on yesterday’s signals — and see why offline/online feature consistency is the system’s spine, not a detail.
Everything you assembled, in order
- Retrieve → rank — cheap-wide recall, then expensive-narrow precision
- Embeddings — taste as geometry — similarity becomes distance
- Two-tower + ANN — live user vector × offline item index, in ms
- Ranking Model — score P(engage) over hundreds with rich features
- Feature Store — offline + online, consistent — no train/serve skew
- Filters & Rules — dedupe, policy, diversity → a shippable feed
- Feedback loop — log engagement, retrain, refresh — the flywheel
- Freshness — stale features rot relevance with no error at all