The big idea
Pick the best few from millions
A user opens the app. Somewhere in a catalog of millions of items are the ten they’d most want to see right now. You have tens of milliseconds to find them. You can’t score millions of items per request — but you can’t guess, either. How do you do this fast and well?
Use a funnel. A cheap, fast stage narrows millions to a few hundred plausible candidates; a heavy, accurate stage ranks those few hundred precisely. Cheap-and-wide, then expensive-and-narrow — the pattern behind every large recommender.
How to read this: Each step opens with a real design decision — you make the call before I show you what ships. Watch the funnel grow, hover any box, replay the flow. At the end stale the features to see relevance quietly rot. Hit Begin.
Step 1 · The skeleton
A request and a funnel
The user opens the app and needs a feed. We can’t run a heavy model over millions of items in the time budget. What overall structure makes "best ten from millions" feasible?
Design decision: You must return the best 10 of millions in ~tens of ms. What’s the shape?
The call: A two-stage funnel: cheap retrieval, then heavy ranking. — Stage 1 narrows millions to hundreds cheaply; stage 2 ranks those hundreds with a heavy model. Each stage is sized for its job.
A Recs Gateway runs a two-stage funnel for every request: candidate generation (millions → hundreds, fast) then ranking (hundreds → ordered, accurate), with filtering before the final list ships to the user. The shape is the whole idea.
Retrieve, then rank: No single model can be both fast over millions and precise per item. Splitting into a recall stage (don’t miss good items) and a precision stage (order them well) is how recommenders hit both latency and quality.
Step 2 · Represent taste
Embeddings for users and items
To find items a user will like, you need to compare a user against items numerically. "Action movies" and "this thriller" need to be close somehow. How do you represent taste so similarity is computable?
Design decision: How do you make "this user" and "this item" comparable for similarity?
The call: Learn embeddings that place similar users and items near each other. — Train embeddings from engagement so a user vector lands near the item vectors they’d enjoy. Similarity becomes distance in that space.
The system learns embeddings: users and items map to vectors in a shared space where a user sits near the items they’d engage with. These come from training on past engagement, so proximity encodes learned taste — not hand-typed tags.
Taste as geometry: Embeddings turn "would this user like this item?" into "how close are their vectors?" Once taste is geometry, retrieval becomes a nearest-neighbour search — fast and scalable.
Step 3 · Narrow the field
Two-tower retrieval with ANN
You have a user vector and millions of item vectors. You need the few hundred closest — in milliseconds, per request. Comparing against every item exactly is too slow. How do you retrieve candidates fast?
Design decision: Find the few hundred closest items to the user vector, fast. How?
The call: Precompute item vectors offline, index them, and run ANN at request time. — A "two-tower" model produces the user vector live and item vectors offline; an ANN index returns the nearest few hundred in milliseconds.
A two-tower model has a user tower (computed live from the request) and an item tower (item vectors precomputed offline and stored in the Item Index). Candidate Generation embeds the user, then runs ANN search to pull the nearest few hundred items in milliseconds. Recall stage: don’t miss the good ones.
Two towers, one space: Item vectors are computed offline (millions, slow, cached); the user vector is computed online (one, fast). They meet in the same space via ANN — decoupling expensive item work from the live request.
Step 4 · Rank with care
The heavy ranking model
Retrieval handed back ~hundreds of candidates, ordered only by rough embedding similarity. Similarity isn’t the same as "will this specific user engage right now." How do you get the order right?
Design decision: You have ~hundreds of candidates. How do you order them precisely?
The call: Score each candidate with a heavy model predicting engagement, using rich features. — A ranking model scores P(engage) for each candidate using many user/item/context features — affordable now because there are only hundreds, not millions.
A heavy Ranking Model scores each candidate’s probability of engagement (click, watch, purchase) using rich features — user history, item attributes, context (time, device), cross features. It’s far too expensive to run over millions, but perfect over a few hundred. Precision stage: order them right.
Why two stages, again: Retrieval is cheap and approximate over millions; ranking is expensive and precise over hundreds. The funnel spends compute only where it pays off — the defining move of scalable recommenders.
Step 5 · Feed the model
The feature store
Ranking needs features — "how many cooking videos did this user watch today?", "this item’s 1-hour click-rate". Some are slow to compute; all must be fresh and identical between training and serving. Where do features come from?
Design decision: Ranking needs fresh features, and training must see the SAME values. How?
The call: A feature store: offline batch pipeline + low-latency online serving, kept consistent. — Compute features in an offline pipeline AND serve them online with low latency, from one definition — so training and serving see identical values (no skew).
A Feature Store serves ranking its features and has two halves that must agree: an offline pipeline computes features in batch (for training), and an online store serves the freshest values at request time (for serving) — from a single definition, so both see the same numbers.
Train/serve skew is the silent killer: If a feature is computed one way in training and another at serving, the model scores on values it never learned from — and quality quietly rots. The feature store exists to make offline and online provably consistent.
Step 6 · The last mile
Filters, rules and diversity
Ranking gives a perfectly-ordered list — but the top items might be things the user already saw, items that violate policy, or ten near-identical videos. A great score isn’t a shippable feed. What sits between ranking and the screen?
Design decision: The ranked list is perfect by score. Why isn’t it ready to ship?
The call: Apply filters: dedupe seen, enforce policy, add diversity and business rules. — A filtering stage removes already-seen/blocked items, enforces policy and freshness, and spreads diversity — turning a scored list into a shippable feed.
A Filters & Rules stage takes the ranked list and makes it shippable: drop already-seen and blocked/policy-violating items, enforce diversity (don’t show ten near-identical items), and apply business rules (freshness, sponsored slots, fairness). Then the final feed goes to the user.
Score ≠ shippable: Pure relevance order ignores what a user has seen, what policy forbids, and how monotonous a list feels. Post-ranking rules encode the product judgment a score can’t.
Step 7 · Close the loop
Log engagement, retrain, repeat
The feed shipped. The user clicks some things, ignores others. That reaction is the single most valuable signal you have — it’s both the truth about what’s good and the fuel for tomorrow’s models. How do you use it?
Design decision: The user reacts to the feed. What do you do with that signal?
The call: Log every impression and interaction, then retrain retrieval + ranking on it. — Engagement events become training data and freshness signals: retrain the models periodically and refresh features so the system keeps improving.
Every impression and interaction is logged to the Engagement Log. A Training Pipeline periodically retrains the retrieval and ranking models on that data and refreshes the feature store — so the system learns from what worked and adapts as taste shifts. The loop is the product.
The flywheel: Recommendations generate engagement, engagement becomes training data, better models generate better recommendations. A recommender is less a model than a self-improving loop — which is exactly why stale features are so damaging.
The payoff
You built a recommender
From "best ten of millions" to a self-improving funnel: two-tower retrieval, a heavy ranker, a consistent feature store, post-ranking rules, and a feedback loop that retrains on engagement.
Now stale the feature store and watch relevance quietly rot — recommendations built on yesterday’s signals — and see why offline/online feature consistency is the system’s spine, not a detail.
- Retrieve → rank — cheap-wide recall, then expensive-narrow precision
- Embeddings — taste as geometry — similarity becomes distance
- Two-tower + ANN — live user vector × offline item index, in ms
- Ranking Model — score P(engage) over hundreds with rich features
- Feature Store — offline + online, consistent — no train/serve skew
- Filters & Rules — dedupe, policy, diversity → a shippable feed
- Feedback loop — log engagement, retrain, refresh — the flywheel
- Freshness — stale features rot relevance with no error at all