The big idea
What is an embeddings service?
Semantic search, RAG, recommendations and clustering all rest on one primitive: turning text (or images) into a vector that captures meaning, so "close in vector space" means "similar in meaning". You need to produce these embeddings for a single query in milliseconds and for millions of documents to build an index — two very different jobs, both GPU-bound.
An embeddings service serves both modes over an embedding model: a low-latency online path for query embeddings and a high-throughput batch pipeline for indexing corpora. The subtle, critical part isn't the model — it's consistency: the query and the entire corpus must be embedded with the same model version, or search silently breaks.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end swap the model version and drop the cache to see the consistency rule and the cost lever. Hit Begin.
Step 1 · Meaning as a vector
Why embeddings, and why a service?
Keyword search matches words, not meaning — "how to reset my password" misses a doc titled "recovering account access". Embeddings fix that by mapping meaning to vectors. So why not just call the model inline wherever you need one?
Design decision: Why build an embeddings *service* instead of calling the model inline everywhere?
The call: Embedding is GPU-bound and used in two very different modes (real-time query + bulk indexing) that need shared, consistent, scalable serving. — Producing embeddings is heavy GPU inference, needed both at low latency for queries and at massive throughput for indexing — and every consumer must use the identical model/version for vectors to be comparable. Centralizing this into a service gives shared batching, scaling, caching and version control that inline calls can't.
Embedding is heavy GPU inference needed in two modes — real-time query embedding and bulk corpus indexing — and every consumer must use the same model version for vectors to be comparable. A shared service centralizes GPU batching, autoscaling, caching, and (most importantly) version control, which scattered inline calls can't guarantee. Meaning-as-a-vector is the primitive; serving it consistently at scale is the system.
Embedding space: An embedding model places semantically similar inputs near each other in a high-dimensional space, so similarity becomes distance (cosine/dot product). Everything downstream — vector search, RAG, clustering — depends on all vectors living in the same space, i.e. from the same model.
Step 2 · Two very different jobs
Online vs batch
A search query needs one embedding, now, with tight latency. Indexing a corpus needs millions of embeddings with maximum throughput and no latency pressure. Why can't one path serve both well?
Split the paths. Online: embed a single query (or a few) with low latency — small batches, fast turnaround, on the search read path. Batch: embed millions of documents for throughput — large GPU batches through a queue, latency-insensitive, to build or refresh the index. Same model, opposite optimization (latency vs throughput), so the API exposes both and routes accordingly — exactly the batch-vs-streaming split you see across AI serving.
Latency path vs throughput path: Online favors small, prompt batches (latency); batch favors big batches (throughput/cost). Conflating them makes queries wait behind bulk jobs or wastes GPU on tiny batches. Separate them so each hits its target — and reserve/prioritize capacity so an indexing run can't starve live queries.
Step 3 · The query path
Online embedding
When a user searches, you must embed their query fast — it's on the critical path before the vector lookup even runs. What does the online path optimize?
The online embedder serves single or small-batch query embeddings with minimal latency: keep the model warm and resident, use small prompt-flushed batches (don't wait to fill a big batch), and consider a smaller/faster embedding model where quality allows. The output is a single vector handed to nearest-neighbor search. This path is latency-critical because it gates every search request, so it's tuned for fast first-response, not throughput.
Warm + small batches: Query embedding is a real-time inference on the search path, so the levers are warm weights (no cold start), tiny batches (latency over utilization), and model size. Even a few tens of milliseconds matter because this runs before retrieval on every query.
Step 4 · Inside the model
Text (or image) → vector
What does the embedding model actually produce, and what properties matter for using its output?
The model maps an input to a fixed-dimension vector (e.g. 384–3072 dims) — for text, typically pooling the transformer's token representations; for images or multimodal, an encoder into the same space. The vector is usually L2-normalized so cosine similarity = dot product. Key properties you must track and keep consistent: the model + version, the dimension, the similarity metric, and any normalization — because every vector compared must share all of them.
Fixed dimension, one space: An embedding is a point in a fixed-dimensional space defined by the model. Dimension, normalization and metric are part of that definition. Two vectors are only comparable if they came from the same model producing the same-dimensional, same-normalized space — which is why versioning (next) is everything.
Step 5 · Index the corpus
The batch embedding pipeline
To power search over millions of documents, you must embed them all and store the vectors — a huge, bursty, GPU-heavy job. How do you run it efficiently?
Design decision: You must embed millions of docs and store the vectors. How do you run it efficiently?
The call: Stream docs through a queue to batched GPU workers that write vectors to the vector store. — Documents flow through a job queue to worker(s) that embed them in large GPU batches (maximizing throughput/cost) and write the resulting vectors to the vector store, incrementally building the index. It's scalable, restartable, and isolated from the low-latency query path.
Run a batch pipeline: stream documents through a queue to GPU workers that embed them in large batches (maximizing throughput and cost-efficiency) and write the vectors to the vector store, incrementally building the index. It's autoscalable on queue depth, restartable (track progress so a failure resumes, not restarts), and isolated from the online path so a giant indexing run can't slow live queries. This is how you turn a corpus into a searchable index.
Queue + big batches + write: Bulk embedding is a classic throughput job: buffer via a queue, saturate GPUs with large batches, and land results in the vector store. Chunk long documents first (models have input limits) and record which model/version produced each vector — you'll need it when the model changes.
Step 6 · The rule that breaks everything
Model versioning & consistency
Here's the AI-specific trap that silently destroys search quality. Your corpus was embedded months ago with model v1. You upgrade to a better model v2 and start embedding queries with it. Suddenly results are garbage — with no error. Why, and what's the rule?
Design decision: Corpus embedded with v1; you upgrade queries to v2. Results turn to garbage. Why?
The call: Vectors from different models live in different spaces and aren't comparable — the query and the whole corpus must use the same model+version, so a model change means re-embedding everything. — Each model defines its own vector space; a v2 query vector and v1 document vectors have no meaningful geometric relationship, so nearest-neighbor distances are nonsense — yet nothing errors. The absolute rule: query and corpus must share the identical embedding model and version, which means upgrading the model requires re-embedding the entire corpus (often via a versioned re-index and cutover).
Vectors from different models live in different spaces and are not comparable — a v2 query against a v1 corpus produces meaningless distances, and nothing errors. So the rule is absolute: the query and the entire corpus must be embedded with the identical model + version. Upgrading the embedding model therefore means re-embedding the whole corpus — typically build a new versioned index in the background, then atomically cut over queries and corpus together. Track the model version on every vector so you never mix them.
One space, one version: This is the operational heart of embeddings at scale. You can't hot-swap the model — it invalidates every stored vector. Re-index under a new version, keep old and new indexes side by side, and switch atomically. Version-tag vectors so a mismatch is detectable, not a silent quality collapse.
Step 7 · Don't re-embed the same thing
Caching & storage
The same query is typed by thousands of users; the same document gets re-indexed unchanged. Embeddings are deterministic for a given (text, model) — so recomputing them is pure waste. How do you avoid it, and where do vectors live?
Cache embeddings keyed by (text, model version) — since the mapping is deterministic, a cache hit skips the GPU entirely (huge for popular queries and unchanged docs on re-index). Deduplicate identical inputs before embedding. Store query embeddings ephemerally (they're cheap to recompute) and document vectors durably in the vector store with their model version and source id. Only genuinely new (text, version) pairs cost a GPU pass.
Deterministic ⇒ cacheable: Embeddings are a pure function of (input, model), so they cache perfectly by that key — and the key must include the model version (a v2 embedding of the same text is a different vector). Dedup and caching are among the biggest cost savers in an embeddings service.
Step 8 · The sharp edges
Chunking, cost, drift & multimodal
Real embedding workloads hit: documents longer than the model's input limit, GPU cost at corpus scale, the ever-present version-drift risk, and multimodal/multilingual needs.
Chunk long documents to fit the model's token limit (with overlap for context) and embed chunks, since retrieval works at chunk granularity anyway. Control cost with batching, caching/dedup, autoscaling, and right-sized models (bigger isn't always worth it). Guard against version drift by version-tagging every vector and refusing to mix versions in one index. Support multimodal/multilingual by choosing models that embed images/text/languages into a shared space when the product needs cross-modal or cross-lingual search. And re-embed incrementally as content changes.
Design for the unhappy path: Long docs → chunk (retrieval is chunk-level). Cost → batch + cache + right-size. Drift → version-tag, never mix. Cross-modal/lingual → a shared-space model. The embedding service's reliability is mostly about consistency and cost discipline — the model is the easy part.
You did it
You just designed an embeddings service.
- Embeddings map meaning to vectors; a shared service gives consistent, scalable GPU serving in two modes.
- Online (single query, low latency) and batch (millions of docs, throughput) are opposite optimizations.
- The online path keeps the model warm with small batches — it gates every search request.
- The model outputs a fixed-dimension, normalized vector; dimension, metric and version define the space.
- A batch pipeline (queue → big GPU batches → vector store) indexes corpora, restartably and isolated from queries.
- The rule that matters most: query and corpus must share the same model+version, so a model change re-embeds everything.
- Cache by (text, version) since embeddings are deterministic; chunk long docs; guard against version drift.