Pick the best few from millions

Use a funnel. A cheap, fast stage narrows millions to a few hundred plausible candidates; a heavy, accurate stage ranks those few hundred precisely. Cheap-and-wide, then expensive-and-narrow — the pattern behind every large recommender.

A request and a funnel

A Recs Gateway runs a two-stage funnel for every request: candidate generation (millions → hundreds, fast) then ranking (hundreds → ordered, accurate), with filtering before the final list ships to the user. The shape is the whole idea.

Embeddings for users and items

The system learns embeddings: users and items map to vectors in a shared space where a user sits near the items they’d engage with. These come from training on past engagement, so proximity encodes learned taste — not hand-typed tags.

Two-tower retrieval with ANN

A two-tower model has a user tower (computed live from the request) and an item tower (item vectors precomputed offline and stored in the Item Index). Candidate Generation embeds the user, then runs ANN search to pull the nearest few hundred items in milliseconds. Recall stage: don’t miss the good ones.

The heavy ranking model

A heavy Ranking Model scores each candidate’s probability of engagement (click, watch, purchase) using rich features — user history, item attributes, context (time, device), cross features. It’s far too expensive to run over millions, but perfect over a few hundred. Precision stage: order them right.

A Feature Store serves ranking its features and has two halves that must agree: an offline pipeline computes features in batch (for training), and an online store serves the freshest values at request time (for serving) — from a single definition, so both see the same numbers.

Filters, rules and diversity

A Filters & Rules stage takes the ranked list and makes it shippable: drop already-seen and blocked/policy-violating items, enforce diversity (don’t show ten near-identical items), and apply business rules (freshness, sponsored slots, fairness). Then the final feed goes to the user.

Log engagement, retrain, repeat

Every impression and interaction is logged to the Engagement Log. A Training Pipeline periodically retrains the retrieval and ranking models on that data and refreshes the feature store — so the system learns from what worked and adapts as taste shifts. The loop is the product.

Design a Recommendation System — A Guided System Design

System Design · step by stepDesign a Recommendation System

Step 1 / 9

Finished this one? 0 / 5 AI System Designs done

More AI System Designs

Design a Conversational AIBuild a production conversational AI system (think ChatGPT) step by step. See how the request path splits an inference gateway from the model servers, how the context window is assembled and token-budgeted, how conversation memory is stored and recalled, how tokens stream back over a persistent connection, and how guardrails gate every prompt and response — through an interactive diagram that grows with each concept.Read →
Design a RAG PipelineBuild a retrieval-augmented generation pipeline step by step. See how documents are chunked and embedded, how a vector store answers semantic search, how two-stage retrieval with reranking finds the best passages, how the prompt is grounded to stop hallucination, and how evals keep a quietly-drifting index honest — through an interactive diagram that grows with each concept.Read →
Design an AI Agent SystemBuild an autonomous AI agent step by step. See how the plan-act-observe loop turns a goal into action, how the model emits typed tool calls, how a sandboxed executor runs them safely, how working and long-term memory fit together, and how budgets and approval gates keep a multi-step agent from running away — through an interactive diagram that grows with each concept.Read →
Design an LLM Inference ServerBuild an LLM inference serving system step by step. See how a request queue absorbs spiky traffic, how the prefill/decode split and continuous batching keep GPUs full, how the KV cache and paged attention make each token cheap, how tensor sharding fits a giant model, and how autoscaling rides demand — all balancing latency against throughput, through an interactive diagram that grows with each concept.Read →

Explore more from Vibe Engines

Handbooks →

Roadmaps →

Prompt Engineering Roadmap

System Designs →