The whole design, in writing
Learn AI system design by building a large-scale recommendation system step by step. An interactive guide covering the two-stage retrieve-and-rank funnel, two-tower embedding retrieval with ANN, a heavy ranking model, the feature store and its offline/online split, filtering business rules, and the feedback loop that keeps recommendations fresh.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Pick the best few from millions
A user opens the app. Somewhere in a catalog of millions of items are the ten they’d most want to see right now. You have tens of milliseconds to find them. You can’t score millions of items per request — but you can’t guess, either. How do you do this fast and well?
Use a funnel. A cheap, fast stage narrows millions to a few hundred plausible candidates; a heavy, accurate stage ranks those few hundred precisely. Cheap-and-wide, then expensive-and-narrow — the pattern behind every large recommender.
What the new pieces do
- Userclient
- A person who opens the app expecting a feed of items they’ll actually like — picked from millions.
Step 1 · The skeleton
A request and a funnel
The user opens the app and needs a feed. We can’t run a heavy model over millions of items in the time budget. What overall structure makes "best ten from millions" feasible?
You must return the best 10 of millions in ~tens of ms. What’s the shape?
Running a heavy model over millions of items per request is impossible in the latency budget. You must shrink the set first.
Popularity is a fine fallback but ignores the individual — it’s not personalization, and engagement suffers. You need per-user relevance.
Stage 1 narrows millions to hundreds cheaply; stage 2 ranks those hundreds with a heavy model. Each stage is sized for its job.
A Recs Gateway runs a two-stage funnel for every request: candidate generation (millions → hundreds, fast) then ranking (hundreds → ordered, accurate), with filtering before the final list ships to the user. The shape is the whole idea.
Why this piece earns its place
The funnel sets a ceiling nothing downstream can lift: an item candidate generation never returns cannot be recommended, however good the ranker becomes. That matters because ranking metrics are computed over the candidate set, so they keep looking healthy while the item a user would have loved was never in the room — a retrieval regression surfaces as a ranking plateau, and a team can spend a quarter tuning the wrong stage. The stages are also less sequential than the drawing suggests. Nothing the ranker needs about the user depends on which candidates come back, so that fetch can start the instant the request arrives and overlap retrieval entirely, while anything about the items cannot begin until there is a candidate list. Which half of the work is blocked on what is usually how the budget gets met, rather than by making any one stage faster. The gateway is also where the per-stage deadlines live, and every stage needs an answer for blowing one. Ranking times out and you can ship the candidate order; filters time out and you ship a shorter feed. What you cannot do is return nothing — unlike a search box, where no results is an honest answer, an empty home screen reads as a broken app.
What the new pieces do
- Recs Gatewaybackend
- Runs the funnel for each request: fetch candidates, rank them, apply rules, return the final list.
Step 2 · Represent taste
Embeddings for users and items
To find items a user will like, you need to compare a user against items numerically. "Action movies" and "this thriller" need to be close somehow. How do you represent taste so similarity is computable?
How do you make "this user" and "this item" comparable for similarity?
Tags are coarse and miss latent taste — two thrillers can feel totally different. Learned embeddings capture nuance hand-tags can’t.
Train embeddings from engagement so a user vector lands near the item vectors they’d enjoy. Similarity becomes distance in that space.
IDs carry no notion of similarity — item 5012 isn’t "near" item 5013. You need a learned space where proximity means relevance.
The system learns embeddings: users and items map to vectors in a shared space where a user sits near the items they’d engage with. These come from training on past engagement, so proximity encodes learned taste — not hand-typed tags.
Why this piece earns its place
Treat the space as a versioned artifact, not a property of a model. Both towers are trained together and have to go live together, which makes this a cutover rather than a sequence: build the new index from the new item tower beside the one already serving, check it, then move queries and the new user tower across in one switch. The new user tower must never address the old index, and the old one must never address the new. Either mismatch has queries from one space searching an index built in another, and nearest-neighbour across two spaces returns real-looking distances over essentially arbitrary items, with nothing anywhere throwing an error. Stamp every index with the model version that produced it so a mismatch is a startup assertion rather than an incident, and keep the previous index warm afterwards, because a rollback has to take both halves back together. The second thing worth knowing is what this space buys you elsewhere. Item vectors are the cheapest similarity you will ever have, and the near-duplicate rules further down the funnel run on those same distances — one artifact, several consumers. That makes its refresh cadence a shared dependency: bump the embeddings to improve retrieval and you have quietly changed what the feed considers repetitive, which is not a connection anyone remembers when the complaint arrives.
- millionsitems, each a vector
- 1 vectorper user
- distance= predicted affinity
Step 3 · Narrow the field
Two-tower retrieval with ANN
You have a user vector and millions of item vectors. You need the few hundred closest — in milliseconds, per request. Comparing against every item exactly is too slow. How do you retrieve candidates fast?
Find the few hundred closest items to the user vector, fast. How?
Exact scoring over millions per request blows the latency budget. At this scale you use an approximate index.
A "two-tower" model produces the user vector live and item vectors offline; an ANN index returns the nearest few hundred in milliseconds.
Category filters are a blunt pre-filter that miss cross-category gems and still leave too many to score exactly. ANN over embeddings is the scalable answer.
A two-tower model has a user tower (computed live from the request) and an item tower (item vectors precomputed offline and stored in the Item Index). Candidate Generation embeds the user, then runs ANN search to pull the nearest few hundred items in milliseconds. Recall stage: don’t miss the good ones.
Why this piece earns its place
The reason this is two towers and not one model is a hard constraint, and naming it is most of the interview. The towers never see each other — the item side was computed hours ago, the user side is computed from the request, and they meet only at a distance. So retrieval structurally cannot use any user-item cross feature: whether this person has watched this creator before, how people in this city reacted to this item in the last hour. Those only become computable once the set is small enough to score pairs, which is why the next stage exists and why you cannot simply move it earlier. Production systems buy back some of that lost expressiveness by running several retrievers rather than one — a recent-interest source, a social-graph source, a trending source — and unioning the results before ranking. That is a deliberately cheap change: each source is an independent index with its own quota of the candidate budget, and adding one never touches the ranker. It also means the candidate count stops being a single number you tune and becomes an allocation you argue about.
What the new pieces do
- Candidate Genservice
- Narrows millions of items to a few hundred plausible candidates fast, using embedding similarity.
- Item Indexindex
- Vector index of every item’s embedding, so candidate generation is a nearest-neighbour lookup.
Step 4 · Rank with care
The heavy ranking model
Retrieval handed back ~hundreds of candidates, ordered only by rough embedding similarity. Similarity isn’t the same as "will this specific user engage right now." How do you get the order right?
You have ~hundreds of candidates. How do you order them precisely?
Embedding distance is a coarse recall signal, not a precise engagement prediction. The top few hundred need a real scoring model.
Popularity ignores the individual and context. The whole point of ranking is per-user, per-moment precision.
A ranking model scores P(engage) for each candidate using many user/item/context features — affordable now because there are only hundreds, not millions.
A heavy Ranking Model scores each candidate’s probability of engagement (click, watch, purchase) using rich features — user history, item attributes, context (time, device), cross features. It’s far too expensive to run over millions, but perfect over a few hundred. Precision stage: order them right.
Why this piece earns its place
Two things about this model only show up once it is running. First, ranking well and predicting well are not the same. A model can order candidates correctly and still be badly calibrated — the number it emits reads as a probability and is not one. Ordering is all you need while the score’s only job is sorting, but the moment anything does arithmetic with it — blending several predictions, comparing against a fixed cutoff, deciding whether a slot is worth filling at all — an uncalibrated score corrupts that arithmetic quietly, and rank-based offline metrics will not notice, because any transformation that preserves order leaves them unchanged. Calibrate against held-out outcomes and re-check after every retrain, since calibration drifts even when ranking quality does not. Second, this is where the compute bill lives. The ranker runs over every candidate on every request, so its cost scales with candidates times traffic, not traffic — far and away the largest inference spend in the design. When it gets too expensive, reach for the size of the candidate set before a smaller model. Both costs are measurable; the difference is reversibility. The candidate count is a number you can turn down on one surface this afternoon and put back tomorrow, while a smaller ranker is a retrain, an evaluation and a release before you know what you gave up.
What the new pieces do
- Ranking Modelservice
- A heavy model that scores each candidate’s probability of engagement using rich user/item features.
Step 5 · Feed the model
The feature store
Ranking needs features — "how many cooking videos did this user watch today?", "this item’s 1-hour click-rate". Some are slow to compute; all must be fresh and identical between training and serving. Where do features come from?
Ranking needs fresh features, and training must see the SAME values. How?
Heavy aggregations can’t be computed within the latency budget per request. And ad-hoc computation drifts from what training saw.
Compute features in an offline pipeline AND serve them online with low latency, from one definition — so training and serving see identical values (no skew).
Features are dynamic (today’s watch count); they can’t be frozen into weights. They must be served live and refreshed.
A Feature Store serves ranking its features and has two halves that must agree: an offline pipeline computes features in batch (for training), and an online store serves the freshest values at request time (for serving) — from a single definition, so both see the same numbers.
Why this piece earns its place
The half of this that nobody mentions until it bites is point-in-time correctness. Training rows are built by joining logged impressions to features, and the value you have to join is the one the online store would have served at that instant — not the one sitting there now. Join current numbers onto last month’s impressions and you have handed the model a feature that already knows the outcome: a user’s watch count for a creator includes the watch you are trying to predict. Offline metrics improve, the online test does not move, and nothing errors. Getting it right means every feature is written with its own event time and read as of a timestamp, which is a far heavier requirement than serving the latest value, and the real reason this is infrastructure rather than a cache. The same requirement decides what happens when a value simply is not there at request time, which the online half will occasionally do. The ranker scores the row regardless, so whatever gets substituted has to be what training substituted — a default picked quietly in serving code is the one skew nobody documents, and it shows up as a model that ranks a small, unlucky slice of users badly while every aggregate metric looks fine.
What the new pieces do
- Feature Storestore
- Serves the user/item/context features ranking needs, with offline (batch) and online (low-latency) halves.
Back of the envelope
- offline half
- batch-compute features → training data
- online half
- serve freshest values at request, low latency
- one definition
- same logic both sides → no train/serve skew
- freshness matters
- today’s behavior must reach ranking today
Step 6 · The last mile
Filters, rules and diversity
Ranking gives a perfectly-ordered list — but the top items might be things the user already saw, items that violate policy, or ten near-identical videos. A great score isn’t a shippable feed. What sits between ranking and the screen?
The ranked list is perfect by score. Why isn’t it ready to ship?
Top-by-score can repeat already-seen items, surface blocked content, and stack near-duplicates. Real feeds need post-ranking rules.
Diversity matters, but blindly maximizing it tanks relevance. It’s one constraint among several applied after ranking, not a replacement for it.
A filtering stage removes already-seen/blocked items, enforces policy and freshness, and spreads diversity — turning a scored list into a shippable feed.
A Filters & Rules stage takes the ranked list and makes it shippable: drop already-seen and blocked/policy-violating items, enforce diversity (don’t show ten near-identical items), and apply business rules (freshness, sponsored slots, fairness). Then the final feed goes to the user.
Why this piece earns its place
The rules here are not all of one kind, and the difference decides what the stage above owes this one. Already-seen and blocked are subtractions: they remove candidates outright, so the candidate budget has to be set from the observed survival rate rather than from the length of the feed. A heavy user has seen most of what retrieval returns, so a few hundred candidates can collapse to a handful and the feed ships short — and the user whose survival rate is unusual needs more candidates, not a thinner screen. Diversity and sponsored slots subtract nothing; they re-order and they reserve, changing which items reach the screen without changing how many you had to retrieve for it. The cheapest structural fix is to push the hard exclusions up into retrieval so the index never returns them, leaving only the order-dependent rules down here. The already-seen set is the awkward state on this path: it has to reflect an impression from seconds ago, because page two of a scroll must not repeat page one, so the impression write sits on the hot path, and it grows per user forever, which in practice means a TTL and an approximate structure. Approximate means it will sometimes suppress an item the user never saw. That is the right trade — just know you made it.
What the new pieces do
- Filters & Rulesservice
- Removes already-seen, blocked, or policy-violating items and applies diversity/business constraints.
Step 7 · Close the loop
Log engagement, retrain, repeat
The feed shipped. The user clicks some things, ignores others. That reaction is the single most valuable signal you have — it’s both the truth about what’s good and the fuel for tomorrow’s models. How do you use it?
The user reacts to the feed. What do you do with that signal?
Ignoring engagement freezes the system — it never learns the user’s shifting taste. The feedback loop is what keeps recs alive.
Engagement events become training data and freshness signals: retrain the models periodically and refresh features so the system keeps improving.
Without impressions you can’t tell "shown and ignored" from "never shown" — you lose the negatives the model needs to learn from.
Every impression and interaction is logged to the Engagement Log. A Training Pipeline periodically retrains the retrieval and ranking models on that data and refreshes the feature store — so the system learns from what worked and adapts as taste shifts. The loop is the product.
Why this piece earns its place
What you log is a schema decision you get to make roughly once. Every model, every feature and every offline evaluation downstream reads this stream, and anything not written at serving time cannot be recovered afterwards: which model versions produced this feed, which retriever the item came from, what position it occupied, what the ranker scored it. Without those columns you can attribute nothing — a metric moves and there is no way to say which of the three things that shipped that week moved it. Adding a field only helps from the day you add it, so an omission costs weeks of history. The second thing is that labels arrive late, and at different speeds. A click is immediate, a completed watch is minutes, a purchase that survives a return window is days. Join the log too eagerly and every slow positive is recorded as a negative, and the model learns to prefer exactly the fast, shallow engagement you were trying to move away from. Each label needs its own waiting period before its row is trainable, and that period — not the retrain schedule — sets how quickly the system can actually learn.
What the new pieces do
- Engagement Logbus
- Every impression and interaction, logged — the training data and freshness signal for the whole system.
- Training Pipelinebus
- Periodically retrains the retrieval and ranking models on logged engagement so they keep improving.
The payoff
You built a recommender
From "best ten of millions" to a self-improving funnel: two-tower retrieval, a heavy ranker, a consistent feature store, post-ranking rules, and a feedback loop that retrains on engagement.
Now stale the feature store and watch relevance quietly rot — recommendations built on yesterday’s signals — and see why offline/online feature consistency is the system’s spine, not a detail.
Everything you assembled, in order
- Retrieve → rank — cheap-wide recall, then expensive-narrow precision
- Embeddings — taste as geometry — similarity becomes distance
- Two-tower + ANN — live user vector × offline item index, in ms
- Ranking Model — score P(engage) over hundreds with rich features
- Feature Store — offline + online, consistent — no train/serve skew
- Filters & Rules — dedupe, policy, diversity → a shippable feed
- Feedback loop — log engagement, retrain, refresh — the flywheel
- Freshness — stale features rot relevance with no error at all