The whole design, in writing
Learn AI system design by building an embeddings service that powers semantic search, RAG and recommendations step by step. An interactive guide covering online vs batch embedding, the model inference path, a batched GPU pipeline that indexes millions of documents, the critical model-versioning/consistency rule, caching and dedup, and the unhappy paths (version drift, cost, chunking, multimodal).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is an embeddings service?
Semantic search, RAG, recommendations and clustering all rest on one primitive: turning text (or images) into a vector that captures meaning, so "close in vector space" means "similar in meaning". You need to produce these embeddings for a single query in milliseconds and for millions of documents to build an index — two very different jobs, both GPU-bound.
An embeddings service serves both modes over an embedding model: a low-latency online path for query embeddings and a high-throughput batch pipeline for indexing corpora. The subtle, critical part isn't the model — it's consistency: the query and the entire corpus must be embedded with the same model version, or search silently breaks.
What the new pieces do
- Clientclient
- Two needs: embed a single query in real time (search), and embed millions of documents to build/refresh an index (batch).
Step 1 · Meaning as a vector
Why embeddings, and why a service?
Keyword search matches words, not meaning — "how to reset my password" misses a doc titled "recovering account access". Embeddings fix that by mapping meaning to vectors. So why not just call the model inline wherever you need one?
Why build an embeddings *service* instead of calling the model inline everywhere?
Producing embeddings is heavy GPU inference, needed both at low latency for queries and at massive throughput for indexing — and every consumer must use the identical model/version for vectors to be comparable. Centralizing this into a service gives shared batching, scaling, caching and version control that inline calls can't.
Embeddings are vectors, not text. The reason for a service is operational: shared GPU serving, batching, caching, and — critically — guaranteeing one consistent model version across all consumers.
Vector size is a model property. A service exists to serve embedding inference efficiently in both modes and to enforce version consistency, not to shrink vectors.
Embedding is heavy GPU inference needed in two modes — real-time query embedding and bulk corpus indexing — and every consumer must use the same model version for vectors to be comparable. A shared service centralizes GPU batching, autoscaling, caching, and (most importantly) version control, which scattered inline calls can't guarantee. Meaning-as-a-vector is the primitive; serving it consistently at scale is the system.
Why this piece earns its place
Start with when you should not build this. One caller, one index, a hosted embedding API behind it, and the service is a proxy with a version string in it — a constant in that caller's own config does the same job and costs nothing to operate. It earns its keep at the second consumer, or the second index. Past that, what it buys is a seam you cannot retrofit. Where the weights actually run — a vendor's API now, your own GPUs later, once per-token pricing stops making sense at corpus scale — becomes one deployment's decision instead of an SDK migration in every repository that ever needed a vector. The same seam is where the spend becomes legible. Embedding arrives as one GPU bill, and without a request that records who asked, you cannot tell that a nightly job re-embedding an unchanged corpus is most of it, or slow that job down without touching anything else. Per-consumer attribution and a quota are a couple of fields while you are building this and close to unrecoverable afterwards, because a bill does not remember who caused it.
Step 2 · Two very different jobs
Online vs batch
A search query needs one embedding, now, with tight latency. Indexing a corpus needs millions of embeddings with maximum throughput and no latency pressure. Why can't one path serve both well?
Split the paths. Online: embed a single query (or a few) with low latency — small batches, fast turnaround, on the search read path. Batch: embed millions of documents for throughput — large GPU batches through a queue, latency-insensitive, to build or refresh the index. Same model, opposite optimization (latency vs throughput), so the API exposes both and routes accordingly — exactly the batch-vs-streaming split you see across AI serving.
Why this piece earns its place
Two paths means two deployments, and most of the payoff is in the second word. They do not scale on the same signal — the online fleet reacts to request rate and tail latency, the batch fleet to how much work is still waiting — and no single autoscaler reads both, so a merged deployment is always sized for whichever workload shouts loudest. They tolerate different hardware, too: batch work is restartable and nobody is waiting on it, so it can sit on capacity that may be reclaimed mid-run, which would be indefensible underneath a live search request. Blast radius differs as well. A bad deploy to the batch fleet delays an index; the same mistake on the online fleet takes search down now. Different rollout caution, different pager. The split also changes the shape of the API, which is easy to miss while it is drawn as one box. Online is text in, vector out. Batch cannot be — a caller that posts a corpus and waits on the connection times out long before the work is done — so that side hands back a job id, and needs status, progress and cancel to go with it. One endpoint, two contracts.
- online1 query, low latency
- batchmillions, high throughput
- same modelboth paths
What the new pieces do
- Embeddings APIgateway
- Splits real-time single-embed requests from bulk indexing jobs, and checks the cache before spending a GPU.
Step 3 · The query path
Online embedding
When a user searches, you must embed their query fast — it's on the critical path before the vector lookup even runs. What does the online path optimize?
The online embedder serves single or small-batch query embeddings with minimal latency: keep the model warm and resident, use small prompt-flushed batches (don't wait to fill a big batch), and consider a smaller/faster embedding model where quality allows. The output is a single vector handed to nearest-neighbor search. This path is latency-critical because it gates every search request, so it's tuned for fast first-response, not throughput.
Why this piece earns its place
Two things about this path only show up once you have run it. The first is that scaling in is not free. Model weights have to be loaded onto a GPU before a replica can answer anything, which is measured in seconds, not in the units the rest of this path is measured in — so a fleet that scales down overnight pays that whole cost into the first morning spike, and the latency you see in the graph is queueing, not inference. You provision for peak and treat the idle headroom as the price of sitting on the search path. The second is that the model is often not where the time goes. A single short query is a small amount of GPU arithmetic wrapped in a request, a tokenizer, a transfer and a response, and at that size the wrapper can outweigh the matrix multiply — so reaching for a smaller model before measuring that split buys you a quality loss and very little latency. And decide now what search does when this path is unavailable, because search has no answer of its own: fall back to keyword retrieval, serve from cache, or fail the request. That is a product decision, and you do not want to be making it during the incident.
- warm modelno cold start
- small batchlatency first
- 1 vector→ ANN search
What the new pieces do
- Online Embedderservice
- Serves single/small-batch query embeddings fast (small batches to keep latency low), on the read path of search.
Step 4 · Inside the model
Text (or image) → vector
What does the embedding model actually produce, and what properties matter for using its output?
The model maps an input to a fixed-dimension vector (e.g. 384–3072 dims) — for text, typically pooling the transformer's token representations; for images or multimodal, an encoder into the same space. The vector is usually L2-normalized so cosine similarity = dot product. Key properties you must track and keep consistent: the model + version, the dimension, the similarity metric, and any normalization — because every vector compared must share all of them.
Why this piece earns its place
Once the vectors are normalized, cosine, dot product and Euclidean distance rank identically — for unit vectors the distance falls exactly as the dot product rises, so they are one decision wearing three names. That freedom is bought by the normalization, not by the metric, and it is worth knowing which one is doing the work: an approximate index is built around a single distance function, so the metric is chosen when the index is created rather than passed with a query, and trading it costs a rebuild. Dimension is the expensive property. It sets what a million vectors cost in memory and how long a search over them takes, it is picked before the corpus exists, and shrinking it later is the same job as changing models. And same model has to mean the same weights and the same inference code, because pooling and truncation live in the serving library — a routine dependency bump can move the space while the version string sits exactly where it was. Pin the serving image beside the weights, and keep a few fixed sentences and their expected vectors as a fixture every deploy checks itself against. Nothing else catches a space that moved quietly.
- fixed dims384–3072
- normalizecosine = dot
- metricpart of the space
What the new pieces do
- Embedding Modeldata
- The model that maps text (or images) to a fixed-dimension vector. Its VERSION must match across query and corpus.
Step 5 · Index the corpus
The batch embedding pipeline
To power search over millions of documents, you must embed them all and store the vectors — a huge, bursty, GPU-heavy job. How do you run it efficiently?
You must embed millions of docs and store the vectors. How do you run it efficiently?
One-at-a-time synchronous calls waste the GPU (tiny batches) and would take forever for millions of docs, while also competing with live queries. Bulk work belongs on a throughput-optimized batch pipeline with large batches.
Loading millions of docs at once is fragile and unbounded. You stream them through a queue and process in large batches, writing results incrementally to the vector store, so it's scalable and restartable.
Documents flow through a job queue to worker(s) that embed them in large GPU batches (maximizing throughput/cost) and write the resulting vectors to the vector store, incrementally building the index. It's scalable, restartable, and isolated from the low-latency query path.
Run a batch pipeline: stream documents through a queue to GPU workers that embed them in large batches (maximizing throughput and cost-efficiency) and write the vectors to the vector store, incrementally building the index. It's autoscalable on queue depth, restartable (track progress so a failure resumes, not restarts), and isolated from the online path so a giant indexing run can't slow live queries. This is how you turn a corpus into a searchable index.
Why this piece earns its place
The GPUs are usually not the bottleneck here — the store is. Writing into an approximate index is not an append; the structure has to absorb every insert, so a fleet tuned to saturate the GPUs will outrun the writer and the queue backs up behind storage rather than the model. Batch the writes as deliberately as you batch the inference, and for a first full build consider constructing the index offline and loading it instead of inserting a corpus row by row. The second surprise is poison inputs. Inside a queue with retries, one document the tokenizer chokes on is redelivered forever, burning the fleet on a page nobody will ever search for. Dead-letter it and carry on; a backfill that skips a handful of documents beats one that stops dead partway through. The other thing one queue hides is that two workloads are sharing it. A full backfill and the trickle of newly published documents are not the same priority, and a single FIFO puts the trickle behind millions of items — a document published this morning is not searchable until the backfill finishes, which is a freshness bug nobody will attribute to the queue. Two queues, or a priority lane, and steady-state indexing keeps its own promise while the big job runs underneath it.
- queuestream millions
- big batchesthroughput
- → vector storebuild the index
What the new pieces do
- Batch Embedderservice
- Embeds documents in large GPU batches for maximum throughput, writing vectors to the store to build the index.
- Batch Queuebus
- Buffers bulk indexing jobs so millions of documents drain through the GPU fleet at a sustainable rate.
- Vector Storedata
- Where document embeddings land (a vector database), later queried by nearest-neighbor search for retrieval.
Step 6 · The rule that breaks everything
Model versioning & consistency
Here's the AI-specific trap that silently destroys search quality. Your corpus was embedded months ago with model v1. You upgrade to a better model v2 and start embedding queries with it. Suddenly results are garbage — with no error. Why, and what's the rule?
Corpus embedded with v1; you upgrade queries to v2. Results turn to garbage. Why?
Each model defines its own vector space; a v2 query vector and v1 document vectors have no meaningful geometric relationship, so nearest-neighbor distances are nonsense — yet nothing errors. The absolute rule: query and corpus must share the identical embedding model and version, which means upgrading the model requires re-embedding the entire corpus (often via a versioned re-index and cutover).
It's not model quality — it's incompatibility. Even a better model produces vectors in a different space that can't be compared to old ones. You must re-embed the corpus with the new model before switching queries to it.
The store is fine; the vectors are simply from two incompatible models. Mixing model versions in one index silently breaks similarity. Re-embed everything under one version.
Vectors from different models live in different spaces and are not comparable — a v2 query against a v1 corpus produces meaningless distances, and nothing errors. So the rule is absolute: the query and the entire corpus must be embedded with the identical model + version. Upgrading the embedding model therefore means re-embedding the whole corpus — typically build a new versioned index in the background, then atomically cut over queries and corpus together. Track the model version on every vector so you never mix them.
Why this piece earns its place
The switch itself should be a name, not a copy. Queries resolve an index name and the cutover moves that name, so nothing races and nothing half-switches — and the same move made backwards is the rollback, which matters because a relevance regression arrives days later in a human's complaint, not in a monitor at the moment you switch. Both indexes exist for as long as a full corpus embed takes, so peak storage and peak embedding spend are roughly doubled across that window. That, rather than taste, is the honest answer to how often you can afford to change models. The thing that decides whether a second migration is even possible is less obvious: keep the exact text you embedded, durably, beside the vector. If the only copy of it lives in the source system, the next upgrade has to re-read a corpus that has moved on since, and what you build is not a re-embedding of the old index but a different corpus in a new space — which destroys the one comparison the migration rests on, because a difference in results no longer tells you whether the model improved or the documents changed underneath you.
- same model+versionquery = corpus
- model changere-embed all
- versioned reindexthen cut over
Step 7 · Don't re-embed the same thing
Caching & storage
The same query is typed by thousands of users; the same document gets re-indexed unchanged. Embeddings are deterministic for a given (text, model) — so recomputing them is pure waste. How do you avoid it, and where do vectors live?
Cache embeddings keyed by (text, model version) — since the mapping is deterministic, a cache hit skips the GPU entirely (huge for popular queries and unchanged docs on re-index). Deduplicate identical inputs before embedding. Store query embeddings ephemerally (they're cheap to recompute) and document vectors durably in the vector store with their model version and source id. Only genuinely new (text, version) pairs cost a GPU pass.
Why this piece earns its place
The key is the exact bytes of the input, which quietly makes whatever runs before the hash — trimming, collapsing whitespace, case folding — part of the key, versioned by nothing. Change that preprocessing in an ordinary deploy and every existing entry stays perfectly valid and becomes unreachable in the same instant: the hit rate goes to zero at once, not gradually, and the fleet behind it meets a corpus-sized miss rate nobody planned for. Which is the second note — size the online fleet for the miss rate you would have without the cache, or a cache outage stops being a cost event and becomes an availability event on the search path. The other trap is letting both paths share one cache. Their access shapes are opposites: queries are a small hot head, a backfill is a long one-shot stream, so plain LRU lets the backfill evict exactly the entries that were earning the hits, and the query hit rate collapses precisely while the expensive job runs. Give the batch path its own space, or let it read without populating. Entries are immutable, so eviction is a cost knob rather than a correctness one — but each entry is a vector, not a short string, so the ceiling arrives sooner than you expect.
- cache(text, version) → vector
- dedupskip identical inputs
- storevectors + version + id
What the new pieces do
- Embedding Cachecache
- Caches embeddings keyed by (text + model version). Since embeddings are deterministic, identical inputs skip the GPU.
Step 8 · The sharp edges
Chunking, cost, drift & multimodal
Real embedding workloads hit: documents longer than the model's input limit, GPU cost at corpus scale, the ever-present version-drift risk, and multimodal/multilingual needs.
Chunk long documents to fit the model's token limit (with overlap for context) and embed chunks, since retrieval works at chunk granularity anyway. Control cost with batching, caching/dedup, autoscaling, and right-sized models (bigger isn't always worth it). Guard against version drift by version-tagging every vector and refusing to mix versions in one index. Support multimodal/multilingual by choosing models that embed images/text/languages into a shared space when the product needs cross-modal or cross-lingual search. And re-embed incrementally as content changes.
Why this piece earns its place
Chunk size and overlap are part of an index's identity, exactly like the model version. Change them and the stored vectors no longer correspond to the passages the new rules would produce, so a half-migrated index holds two different segmentations of the same corpus and retrieval quietly prefers whichever one happens to be shorter. Tweaking chunking is a full backfill wearing a smaller hat — version it in the same field the model version lives in, and cut over the same way. The other sharp edge is deletion. Vectors outlive their sources: a page is unpublished, a document is redacted, and nothing in the embedding path is watching, so the chunks sit in the index and keep being returned to people who should not see them. Store the source document id on every chunk and make deletion a first-class job rather than something the next full re-index eventually cleans up. And pick the model before the corpus exists. Right-sizing reads like a tuning exercise, but the cost of being wrong is one more full re-embed, so a bake-off on a sample with real queries is cheap now and expensive to postpone.
You did it
You just designed an embeddings service.
Everything you assembled, in order
- Embeddings map meaning to vectors; a shared service gives consistent, scalable GPU serving in two modes.
- Online (single query, low latency) and batch (millions of docs, throughput) are opposite optimizations.
- The online path keeps the model warm with small batches — it gates every search request.
- The model outputs a fixed-dimension, normalized vector; dimension, metric and version define the space.
- A batch pipeline (queue → big GPU batches → vector store) indexes corpora, restartably and isolated from queries.
- The rule that matters most: query and corpus must share the same model+version, so a model change re-embeds everything.
- Cache by (text, version) since embeddings are deterministic; chunk long docs; guard against version drift.
