The whole design, in writing
Learn AI system design by building a semantic search engine step by step. An interactive guide to the embeddings pipeline — a shared embedding model, chunk→embed→index ingest, query embedding, hybrid lexical + vector fusion, cross-encoder reranking, and the model-version-skew trap — so a query finds documents by meaning, not just keywords.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why search by meaning, not words?
Keyword search matches tokens: query "cancel my plan" only finds documents containing those exact words. It misses "end your subscription," "close account," "stop billing" — the same intent in different words, and it ranks a page that merely mentions "plan" ten times above the one that answers the question. The user thinks in meaning; the index thinks in strings. How do you close that gap?
Semantic search embeds text into vectors where nearby = similar meaning, then retrieves by vector distance instead of word overlap. The whole system is an embeddings pipeline: encode the corpus once, encode each query, and find the closest documents in that shared meaning-space.
Step 1 · The skeleton
Text query in, ranked documents out
A client sends a plain-text query and wants the most relevant documents back, ranked. What sits between the words and the answer?
What’s the minimal shape of a semantic-search request?
Substring matching is keyword search’s weakest form — it needs the literal words present and can’t rank by relevance or meaning. It’s exactly what we’re trying to beat.
A coordinator embeds the query into the same space as the documents, searches an index for the closest vectors, and returns ranked ids with scores and snippets.
That’s neither scalable nor cheap — reading the whole corpus per query is O(N) LLM calls. Retrieval narrows to a handful first; the model (if any) comes after.
A Search API takes a text query, turns it into a vector, searches for the nearest document vectors, and returns ranked ids + scores + snippets. Everything else — the embedder, the indexes, the store — hangs off this one contract.
Why this piece earns its place
Everything added later in this build — a second index, a fusion rule, a reranker — arrives behind this one contract without a single caller changing. That is the whole argument for the seam. Let a client query the vector index directly and your retrieval strategy becomes public API: adding lexical search later means shipping a coordinated change to every consumer, so you quietly never do it. Two details in the contract are worth deciding now. If you return scores alongside ids, name the stage that produced them in the same response: the field survives fusion and reranking, the number in it does not mean the same thing afterwards, and a caller that hardcoded a cutoff against the first version of it breaks without saying so. Second, treat pagination as a design decision rather than an offset. Page two of a fused, reordered list is not page one re-sliced, and the cheap answer is to retrieve deep once and paginate inside that shortlist. The other thing only this component can do is log the query beside the ids it returned. That log is the only raw material you will ever have for judging whether the ranking is any good.
What the new pieces do
- Clientclient
- A user or service with a natural-language query — "how do I cancel a plan?" — that wants the most relevant documents ranked by meaning, not exact words.
- Search APIbackend
- Accepts a text query, embeds it, plans retrieval across the vector and lexical indexes, fuses and reranks the candidates, and returns ranked results with scores and snippets.
Step 2 · The shared model
One embedding model, two callers
Both documents and queries have to become vectors, and their vectors are only comparable if they live in the same space. Where does that mapping come from — and who is allowed to use it?
How do documents and queries end up in a comparable vector space?
Different models produce different, incompatible spaces — a query vector from model B can’t be compared to a document vector from model A. Speed can’t justify breaking comparability.
Hashing scatters similar text to unrelated points — the opposite of what you want. Embeddings are learned so that meaning, not spelling, decides nearness.
A single model maps text to a fixed-length vector where nearby = similar meaning. Both the ingest pipeline and the query path call the same model+version, so their vectors are directly comparable.
A single Embedding Model — one set of weights, one version — is the heart of the system. The ingest pipeline calls it to embed documents; the Search API calls it to embed queries. Same model, same space, comparable vectors. Pin that version: it’s the contract every stored vector depends on.
Why this piece earns its place
"The same model" is stricter than it sounds, and the part teams drop is the input convention. Many embedding models are trained asymmetrically and expect the text to be marked — a query embedded as a query, a passage as a passage, often a literal prefix on the string. The ingest path and the query path get written months apart by different people, so one of them forgets, and the result is a quieter version of the failure this page ends on: no exception, slightly worse rankings, nothing to grep for. Put the prefix inside one shared client rather than at two call sites. The second thing to settle is that one model is not one deployment. Ingest calls it in bulk and cares only about throughput; the query path calls it once per request and cares only about tail latency. Share capacity and a backfill will starve live search. Same weights, same version, separate pools. And the dimension is a cost decision you are making here on behalf of everything downstream: every stored vector carries it, and memory in the index scales with it, so halving it later is the same size of migration as changing the model outright.
- 1 modelshared by ingest + query
- fixed dime.g. 384–1536
- version-pinnedvectors depend on it
What the new pieces do
- Embedding Modelmodel
- A single shared model that maps text to a fixed-length vector where nearby = similar meaning. The same model+version must embed both documents (at ingest) and queries (at search) — otherwise the vectors aren’t comparable.
Step 3 · Build the index
Chunk → embed → index the corpus
A document can be a 40-page manual — too big to embed as one vector or return as one result. And the same chunk has to be findable, rankable, and displayable. How does the write path work?
A new batch of documents arrives. What does ingest do with them?
A single vector for 40 pages averages away every specific answer — retrieval gets vague, and you can’t show the user which passage matched. Chunking is what makes results precise.
The pipeline chunks and cleans, embeds each chunk with the shared model, and the indexer upserts it everywhere: vector → ANN index, text+metadata → document store, tokens → lexical index — all keyed by one id.
Embedding at query time makes every search O(N) model calls — hopeless. The corpus is embedded once, offline; queries reuse that work.
The ingest pipeline chunks and cleans each document, embeds every chunk with the shared model, and the Indexer upserts it under one id into three places: the Vector Index (for semantic search), the Document Store (text + metadata to display), and — soon — the Lexical Index. Do it once, offline; serve it forever.
Why this piece earns its place
Three writes, three systems, and no transaction spanning them, so the design question is which half-written chunk you can live with. A vector that lands before its text is an id that ranks, wins a slot in the top five, and hydrates to nothing — the caller quietly gets four results back and no counter anywhere records that it happened. Text that lands before its vector is merely not findable yet, which is indistinguishable from not ingested yet and costs nobody anything. Order the writes by that asymmetry: text and metadata first, the indexes after. Then drive ingest off a durable queue with per-chunk retries, so a failure partway through a large document resumes where it stopped. Restarting the document instead is not just slower — embedding is the expensive half of this path, and you pay for every chunk that already landed a second time. Then accept that ordering is not enough and write a reconciliation pass: count ids in the store against ids in each index and repair the difference on a schedule. It is an afternoon of work and it is the only thing that will ever tell you the three copies have drifted apart, because no query will.
- chunkpassage-sized, coherent
- embed onceoffline, reused
- 1 id → 3 storesvector · lexical · text
What the new pieces do
- Document Storestore
- The chunk text, source, and metadata keyed by id. The indexes return ids + scores; the store hydrates them into snippets to show the user.
- Documentsstore
- The source content — docs, articles, tickets, product pages. Semantic search makes this searchable by meaning; it doesn’t generate anything.
- Chunk & Cleanbus
- Splits documents into passage-sized chunks, strips boilerplate, and normalizes text so each chunk is a coherent unit to embed and return.
- Indexerbus
- Writes each chunk everywhere it belongs: its vector into the ANN index, its tokens into the lexical index, and its text+metadata into the document store — under one shared id.
Back of the envelope
- chunk on structure
- headings/paragraphs beat fixed-size cuts
- overlap a little
- so answers spanning a boundary survive
- batch embeds
- amortize model calls over many chunks
- idempotent upsert
- re-ingesting a doc replaces, not duplicates
Step 4 · The query path
Embed the query, search the space
The index is built. A query arrives as text. Walk the path from words to ranked documents — where does the query become a vector, and what does it hit?
What’s the read path for a semantic query?
The Search API embeds the query into the same space as the documents, asks the ANN index for the nearest chunk vectors, then pulls their text from the document store to return.
That’s keyword matching again — it never uses the embeddings you built. The whole point is to compare vectors, not strings.
The vector index only understands vectors. The query must be embedded first — by the same model that embedded the documents — before it can be compared.
The Search API embeds the query with the shared model, sends the vector to the Vector Index for approximate-nearest-neighbour search, gets back the closest chunk ids + scores, and hydrates them from the Document Store. That’s semantic retrieval end to end — meaning in, ranked documents out.
Why this piece earns its place
Two things go wrong on this path and they deserve opposite responses. An unreachable index is a hard failure — visible, alarmable, and everything above it knows to show an error. A slow or rate-limited embedder is different: with no query vector there is nothing to search with, whatever the index is doing. Worth noticing while the diagram is still this small, because the next step quietly buys the way out — once a second index exists, an embedder outage can degrade to keyword results instead of an empty page, and that becomes a per-request decision the Search API gets to make.
The other thing to be honest about is the word approximate. The index does not promise the true nearest neighbours; it trades a slice of recall for speed, and a neighbour it skipped raises no error, costs no latency and leaves no gap — the list simply comes back one good document shorter. That much you can measure without labelling anything: take a sample of the stored vectors, brute-force the exact neighbours for a batch of real queries, and compare the two lists. No ground truth needed, only vectors you already have. And keep the search-effort knob (ef_search, nprobe) in the request rather than baked into the build, so one expensive query class can buy recall back without re-tuning the index for everybody.
- 1 embedper query
- ANNsub-linear vector search
- hydrateids → snippets from store
What the new pieces do
- Vector Indexindex
- An approximate-nearest-neighbour index (HNSW/IVF) over the document embeddings. Answers "closest vectors to the query vector" in milliseconds — the semantic half of retrieval.
Step 5 · Cover the blind spot
Hybrid search — add lexical BM25
Pure semantic search has a weakness: it blurs exact tokens. Search "error ORA-00942" or the product name "Zephyr-9" and embeddings may return things that are about errors or breezes — semantically near, literally wrong. How do you keep meaning and exact matches?
How do you fix embeddings missing exact terms and rare tokens?
Embeddings compress meaning; they’ll never reliably pin every SKU, error code, or rare name. This is a job for exact matching, not more model.
Vector search rarely returns nothing — it returns something semantically-near-but-wrong. A fallback never triggers; you need both signals on every query.
Hybrid search queries a lexical (BM25) index and the vector index in parallel, then merges their rankings — often with Reciprocal Rank Fusion — so exact tokens and semantic matches both surface.
Add a Lexical Index (BM25) fed by the same indexer. Every query now runs both: vector search for meaning, BM25 for exact terms. The Search API fuses the two ranked lists — typically Reciprocal Rank Fusion (RRF), which needs no score calibration — into one list that has the best of both.
Why this piece earns its place
The lexical index arrives with a configuration surface the vector index does not have, and it decides whether this step does the job you added it for. BM25 only ever sees what its analyzer hands it, and a default analyzer lowercases, strips punctuation, splits on hyphens and stems word endings — which means the product code and the error string you built this half of the system to catch can be shredded into fragments before they are ever indexed. Verify it directly: index a document, search its rarest literal token, confirm the exact string comes back. The semantic side will keep returning plausible-looking results either way and hide the fact that the lexical side is contributing nothing at all. Fusion then adds a parameter nobody writes down: how deep each list runs before the merge. RRF can only see a document’s rank inside a list it appears in, so a chunk past the lexical cut is, to the merge, absent from it — set that cut too shallow and the exact-match signal only fires where the vector search already agreed. And removals run the sequence backwards: clear both indexes first, then the text. Deletes are the one operation here with no self-correcting path. A chunk that never became findable gets noticed the next time somebody searches for it; a chunk that should be gone is noticed only by the person who asked you to remove it.
- BM25 + vectorrun in parallel
- RRFfuse without score calibration
- hybridexact tokens + meaning
What the new pieces do
- Lexical Indexindex
- An inverted-index full-text search (BM25). Catches exact terms, rare tokens, product codes and names that embeddings blur together — the lexical half of hybrid search.
Back of the envelope
- fuse by rank
- RRF avoids incomparable score scales
- weight if needed
- tilt toward lexical for code-heavy corpora
- same chunk ids
- lets the two lists merge cleanly
- dedupe on merge
- a chunk can appear in both lists
Step 6 · Precision at the top
Rerank the shortlist with a cross-encoder
Retrieval gives you ~50 decent candidates fast, but the order of the top 5 is what the user sees. Bi-encoder vector scores are cheap but coarse. How do you sharpen just the top of the list without reranking the whole corpus?
How do you get the ordering of the top results right?
A longer list doesn’t fix order — the right answer buried at rank 30 is as good as absent. You need to reorder the top, not lengthen the tail.
A cross-encoder scores true relevance by attending over the query and document jointly — far more accurate than independent embeddings. It’s slow, so you run it only on the ~50 fused candidates.
A cross-encoder is orders of magnitude slower than ANN — running it over millions of docs per query is impossible. That’s exactly why retrieval narrows to a shortlist first.
A Reranker (cross-encoder) takes the fused shortlist and scores each candidate by reading the query and document together, then reorders the top results. It’s expensive, so it runs on tens of candidates — not the index. Retrieve wide and cheap; rerank narrow and precise.
Why this piece earns its place
Notice what just changed structurally: the Document Store moved. It used to be the last hop of a response, read once to dress a handful of winners as snippets. Now it is a synchronous multi-get of tens of rows before the ranking is even decided, so its read volume scales with the width of the shortlist rather than with what the user sees, and its latency and its availability have become search’s latency and availability. Keep chunk text on a genuinely fast read — a key lookup, not a join across tables — and pull the whole shortlist in one round trip, or one slow row becomes fifty chances to be slow. The detail that decides whether the step earns its cost is truncation. A cross-encoder has a fixed input length and the query and the candidate have to share it, so a long chunk gets cut and what actually gets scored is the opening of the passage rather than the passage. Nobody sized the chunks against that limit — they were sized for retrieval and for display. An answer sitting at the end of a long chunk is invisible to the component you added specifically to find it.
- ~50 → top 5rerank the shortlist
- cross-encoderquery + doc jointly
- precisionthe visible order
What the new pieces do
- Rerankermodel
- A slower cross-encoder that reads the query and each candidate together and scores true relevance. Applied only to the fused shortlist to reorder the final top results.
Step 7 · The silent trap
Model versioning & reindexing
Six months in, a better embedding model ships. You point the query embedder at it and deploy. Search quality craters — but nothing errors, no logs, no empty results. Every stored vector was built by the old model. What just happened?
You upgrade the query embedder but not the corpus. What breaks?
A better model in isolation is better, but the corpus vectors are still from the old model. New-model query vectors and old-model document vectors live in different spaces; comparing them is meaningless.
Speed is unaffected — ANN runs the same. It’s correctness that silently dies: the index returns k ranked results that are near-random.
Embeddings only compare within one model+version. Change the query model without re-embedding the corpus and distances become noise — k results returned, ranked, with no error, but irrelevant.
Query and document vectors are only comparable if they come from the same model+version. Upgrading a model means re-embedding the entire corpus into a new index, then cutting over atomically — often building the new index alongside the old and swapping. Until then, pin the query path to the version the index was built with. Never mix.
Why this piece earns its place
Plan this as a window, not a switch. A backfill over a real corpus runs for hours or days, and documents keep changing while it runs, so the new index is stale before it is finished unless ingest dual-writes into both indexes for the entire window. Cutover is safe only once the new side has caught up, and "caught up" needs a real measurement — the age of the oldest change not yet applied, not a percentage complete, which tells you nothing about the tail. Plan the way back, too. You will not learn that the new model is worse from an alarm, because ranking quality does not raise one; you learn it from complaints, days later, by which point the old index is the thing somebody deleted to reclaim the storage. Keep it until you have evidence, not until the migration script exits. Last, treat reindexes as a scheduling problem, and keep the two kinds apart. A chunking change or a normalization fix alters the text being encoded, so each one is a full pass back through the model. Adding a metadata field to every vector is not — the text is unchanged, so it is a payload rewrite you can backfill without paying for inference. What they share is the expensive machinery around them: a second index, a dual-write window, a cutover. Keep a list, and spend that machinery once.
- same versionquery == index, always
- upgrade = reindexre-embed the whole corpus
- atomic swapbuild new, cut over
Back of the envelope
- stamp vectors with version
- so skew is detectable, not silent
- dual-index on migration
- build v2 beside v1, then swap
- pin query embedder
- to the index’s version until reindex
- backfill in background
- re-embed without downtime
Step 8 · Serve it at scale
Cache, batch, and stay fresh
Popular queries repeat constantly, the embedding model is the most expensive hop per query, and the corpus keeps changing. How do you serve high QPS cheaply without going stale?
What’s the cheapest safe way to cut per-query cost?
The same queries recur, so caching their embeddings — and even fused results with a short TTL — skips the model’s cost. Batching concurrent embeds keeps the model’s throughput high.
The corpus changes — indefinite caching serves stale rankings after new docs land. Result caches need a short TTL or index-version invalidation.
That trades away the precision you added in step 6. The expensive-but-optional hop to cut first is the embedding of repeat queries — via caching — not relevance.
Put a cache in front of the embedding model (keyed by query text + model version) and optionally on fused result sets with a short TTL. Batch concurrent embed calls to keep the model’s throughput up. Invalidate result caches on index updates so fresh documents surface. Now the model runs mostly on new text, and QPS scales.
Why this piece earns its place
Two consequences of caching are easy to miss. The first is that a hit rate is the most flattering number in this design and the wrong one to plan capacity against: it moves your average and not your ceiling, because every phrasing you have not seen before still pays the full embed. Size the embedding fleet against the miss rate, never the request rate. A cache that quietly stops working, or a traffic mix that drifts toward unseen wording, then arrives as an overloaded model rather than as a cache alert. Normalize the text before you key on it, too: trimming and collapsing whitespace and case is the difference between a working cache and one storing a separate entry per typist. The second is that batching is not free latency. Holding a request while a batch fills adds a wait that lands entirely in the tail, so the real knobs are a maximum wait and a maximum size, and under light traffic the window should do nothing at all. A batch window tuned during a load test will sit there adding delay to every query at three in the morning.
- cachequery embeds + hot results
- batchembeds under load
- TTL / invalidateon index change
The payoff
You built semantic search
From "keywords miss meaning" to a system that finds documents by intent: a shared embedding model, a chunk→embed→index ingest path, a query path that embeds and ANN-searches, hybrid BM25 fusion for exact terms, a cross-encoder reranker for precision, and version discipline so the space never fractures.
Now skew the model — upgrade the query embedder without reindexing — and watch relevance collapse with no error at all: k ranked results that are quietly random, because query and document vectors no longer share a space. That’s why the embedding model version is part of the contract, stamped on every vector, and an upgrade is a full reindex.
Everything you assembled, in order
- Search API — one contract: text query → ranked, hydrated documents
- Shared embedding model — same model+version embeds both docs and queries
- Chunk → embed → index — build the corpus once, offline, under one id
- Query path — embed the query, ANN-search, hydrate from the store
- Hybrid search — BM25 + vector, fused by RRF — exact tokens + meaning
- Reranking — a cross-encoder sharpens the top of the shortlist
- Version skew — mismatched models = silent, near-random results
- Cache & batch — the embedder is the bottleneck; skip repeat work