Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
RAG vs long context: retrieve, or paste it all in?
A 1M-token context window makes it tempting to skip retrieval entirely and just paste your whole knowledge base into the prompt. Sometimes that's the right call. But bigger context doesn't mean the model reads it evenly, and it doesn't mean it's cheap — which is exactly what this comparison is about.
The one distinction that decides everything
RAG selects a small, relevant slice of a large corpus at query time via search, and puts only that slice in the prompt. Long context skips selection entirely — you paste a large chunk of (or the entire) corpus directly into the prompt and let the model's attention mechanism find what matters on its own.
If your corpus is bigger than fits comfortably in a context window — or changes often enough that re-sending it every call is wasteful — you need retrieval. If your corpus is fixed and modest, and the task genuinely requires reasoning ACROSS the whole thing at once, long context wins outright.
Head to head
| Dimension | RAG | Long Context |
|---|---|---|
| Corpus size | Effectively unbounded (index scales) | Bounded by the context window, however large |
| Cost per query | Low — small retrieved slice + short prompt | High — pays for prefill over every token sent, every call |
| Freshness | Instant — update the index | Must re-send (and often re-cache) the whole corpus |
| Precision on a needle | High if retrieval finds the right chunk | Degrades — "lost in the middle" attention dilution |
| Cross-document reasoning | Limited to top-k retrieved chunks | Strong — model sees everything jointly, at once |
| Citations | Natural — cite the retrieved chunk | Possible but not built-in, needs extra prompting |
| Setup complexity | Chunking + embedding + index pipeline | None — paste text in |
| Failure mode | Bad retrieval = model never sees the answer | Dilution = model sees it but under-weights it |
When to use each
Reach for RAG
- Corpus is large, unbounded, or grows continuously
- Content updates often — docs, tickets, prices
- Cost per query matters at real request volume
- You need citations back to a specific source
- Most of the corpus is irrelevant to any given query
Reach for long context
- Corpus is fixed-size and fits the budget (a contract set, a codebase)
- The task needs holistic reasoning across everything at once
- You want zero retrieval infrastructure
- The same context is reused across many calls (prompt caching amortizes it)
- Few-shot examples or a whole reference doc, not a huge KB
The answer is usually "retrieve, then go long"
The strongest production pattern isn't RAG or long context — it's RAG feeding a long context window. Instead of retrieving only the top 3 chunks (which can lose cross-document connections), retrieve a broader top-50 or top-100 candidate set and let the model reason over all of them jointly in one long-context call. Retrieval narrows an unbounded corpus down to a size that fits; long context then lets the model actually connect the pieces instead of reasoning over 3 isolated fragments.
Start with RAG at a small top-k — it's the cheapest, fastest option and solves most lookup-shaped questions. Widen the retrieved set and lean on long-context reasoning only when you've seen retrieval miss cross-document connections a small top-k can't hold at once.
Why bigger context doesn’t mean evenly-read context
It's tempting to assume a model with a 1M-token window "reads" all 1M tokens the same way it reads a short prompt — but empirically, that's not how attention behaves. Research on long-context recall (the "lost in the middle" finding) shows models are reliably strong at attending to information near the start and end of a context, and measurably weaker at recalling a fact buried in the middle of a long document, even though nothing was hidden and the token was technically "in the prompt." The context window is a hard ceiling on what the model CAN see; it says nothing about how evenly the model actually weighs what it sees.
This is structurally different from RAG's failure mode. When RAG fails, it's usually because the retriever never fetched the right chunk — a visible, debuggable failure you can fix by tuning the retriever. When long-context recall fails, the right fact WAS in the prompt the whole time, and the model simply under-weighted it relative to information sitting nearer the edges — a much quieter, harder-to-detect failure, because from the outside a plausible-but-wrong answer looks identical to a well-reasoned one.
RAG trades "did I retrieve the right thing" (visible, tunable) for long context's "did the model actually weigh what I gave it evenly" (invisible until you specifically test for it with needle-in-a-haystack evals).
A worked scenario: legal contract review vs a support knowledge base
A tool reviewing a single 200-page merger contract for conflicting clauses is a long-context problem: the corpus is fixed, modest, and the whole point is cross-clause reasoning — clause 14 might contradict clause 187, and RAG's top-k retrieval could easily never place those two chunks in the same call together. Pasting the whole contract into a long-context window (with prompt caching amortizing the cost across the many questions asked about that one contract) is both simpler and more correct here.
A customer-support bot backed by a million-document, hourly-updated knowledge base is the opposite case: the corpus vastly exceeds any context window, most of it is irrelevant to any single question, and updates need to be live within minutes, not require re-sending gigabytes of docs. RAG's index-and-retrieve model is the only one of the two that scales here — and if a question genuinely needs to synthesize across a handful of related articles, that's exactly the "retrieve-broad-then-go-long" combination from section 04, not a reason to abandon retrieval.
Fixed + small + needs holistic reasoning → long context. Unbounded + changing + mostly-irrelevant-per-query → RAG. Most real systems have both shapes of problem inside them, at different points.
The flip point: the fraction of the corpus each query needs
Long context re-sends the whole document set every call, so cost scales with total corpus size D; RAG retrieves only the slice a query actually needs, k tokens, so it sends roughly k/D of the tokens per query.
With a 200K-token corpus where a typical question needs about 2K tokens of it, RAG puts ~2K/200K = 1% of the tokens in each prompt — the rest is retrieval overhead, not model tokens. So the flip is the fraction each query needs: a large corpus queried in small slices → RAG wins big on cost and on “lost in the middle” recall; a small corpus, or questions that genuinely reason over the whole thing → long context wins, especially once prompt caching makes re-sending the fixed corpus cheap. Estimate your own k/D: the smaller it is, the more RAG pays off.
The picture: what each approach puts in front of the model
Cost and trade-offs: tokens, latency and change
Here Time is latency or time to apply a change, and Space is tokens sent or stored. The 200K corpus and 2K slice are the page’s own numbers. The 1,000 questions a day and 400-token chunks are illustrative.
| Case | Time | Space | Why |
|---|---|---|---|
| Input per question | Long: prefill runs over 200K tokens before the first output token. RAG: one search, then a ~2K prefill | 200K vs ~2K tokens: 100× more per call | Time to first token grows with prompt length. The long prompt is paid in latency as well as tokens. |
| 1,000 questions a day | Same per-call latency, every call | 200,000,000 vs 2,000,000 input tokens a day | Caching the fixed corpus cuts the price per token, not the count. Cached tokens are still billed and still processed. |
| Edit one document | Long: rebuild the prompt; a prefix cache is invalid from the edit onward. RAG: re-embed the changed chunks | Long: the whole corpus is re-sent at least once. RAG: a few index rows | An edit near the start of a cached prefix throws away the cache for everything after it. |
| Corpus grows 10× to 2M tokens | Long: past a 1M-token window, it no longer fits. RAG: indexing takes longer; queries do not | RAG prompt stays ~2K | RAG’s cost per query depends on k, not D. That is why it is the only route for unbounded corpora. |
| Question needs 30 passages | RAG top-5 (5 × 400 = 2,000 tokens) cannot hold them | Top-50 is 50 × 400 = 20,000 tokens, still 10% of the corpus | The page’s retrieve-then-go-long pattern: widen k before abandoning retrieval. |
| Per-user access | RAG: filter chunks by the caller’s permissions before the prompt is built | Long: one shared corpus means every caller’s prompt holds every document | Mixed permissions force long context into a separate corpus per access group, and a separate cache for each. |
| A 10-turn conversation | Long: each turn re-sends the corpus plus the growing history. RAG: retrieve again each turn | Long: 10 × 200K corpus tokens if nothing is cached | Follow-ups like “what about the second one?” need RAG to rewrite the query from the history before searching. |
| Find out why it was wrong | RAG: log the retrieved chunk ids; was the answer there? Long: run needle tests at several depths | Eval set of questions with known answer locations | RAG’s failure is visible per query. Long context’s failure only shows up in a positional eval. |
Every row comes back to k/D. Long context costs scale with D, the corpus, on every call. RAG costs scale with k, the slice, and RAG adds one new risk: retrieval. When D is small and fixed, and questions use most of it, the two converge, and long context’s simplicity wins. When D is large or changing, or callers see different documents, nothing in this table favours sending all of it.
Common mistakes
- Testing long context only at the edges
A needle-in-a-haystack test with the fact on page 1 passes, and so does one with it on the last page. Neither tells you about the middle. Place the same known fact at several depths (10%, 50%, 90%) and compare recall, using your real document types rather than a synthetic filler text. Re-run it when the model or the typical prompt length changes, because positional behaviour differs between models and between lengths.
- Treating a cached prompt as free
Prompt caching lowers the price of the repeated prefix. It does not make those tokens disappear: they are still billed at the cached rate, and caches expire and are invalidated by edits. Do the per-day arithmetic with caching on and with a cold cache, because the first call after an edit pays full price.
- Top-3 retrieval for questions that synthesise
“Which of our policies conflict?” needs many passages at once, and three chunks guarantee a partial answer that sounds complete. Route synthesis questions to a wider retrieval (top-50, or every chunk from the matching documents) and keep small k for lookups.
- Burying the question under the documents
In a long prompt, position matters. Put the documents first and the question and instructions last, and put the most relevant material near the start or the end rather than the middle. This works with the positional effect the page describes instead of against it.
- Judging RAG only by its final answers
When a RAG answer is wrong, either the retriever missed or the model misread a good chunk. Those need opposite fixes. Score retrieval separately: for a labelled question set, how often is the answer-bearing chunk in the top k? Without that number, every prompt change is a guess. For a single bad answer, paste the right chunk into the prompt by hand. If the answer becomes correct, retrieval was the problem.
Which would you pick?
Three situations. Pick the side you'd actually build — the explanation follows.
A 40-page contract, fixed for the length of the conversation, where questions need reasoning across the whole document.
Fixed, small and requiring reasoning across the whole thing is the long-context case exactly. Chunking a document whose meaning is distributed across clauses risks retrieving the clause that matches the words while missing the one that changes what it means — and a very large top-k is long context with extra retrieval machinery in front.
A corpus of millions of documents where any one question touches a handful of them.
No context window holds millions of documents, and even where one nearly could, you would be paying to re-send an overwhelmingly irrelevant corpus on every call. This is precisely the shape retrieval exists for.
What does long context give up compared with RAG, beyond cost?
RAG's failure mode is "did I retrieve the right thing", which is visible and tunable. Long context replaces it with "did the model actually attend to all of this", which produces a confident answer either way and only surfaces if you deliberately test for position effects. "Always ignores the middle" overstates a real effect into a law.
Quick answers
RAG or long context — which should I use?
Decide by corpus shape. If it’s large, unbounded, or updates often, use RAG. If it’s fixed-size, fits the budget, and the task needs holistic reasoning across the whole thing, use long context. Many production systems retrieve a broad candidate set and then reason over it in a long-context call.
Does a bigger context window make RAG unnecessary?
No. A bigger window raises the ceiling on how much you CAN paste in, but it doesn’t fix "lost in the middle" attention dilution, and it doesn’t fix the cost of re-sending a huge prompt on every single query. For unbounded, frequently-updated corpora, RAG stays necessary regardless of window size.
What is "lost in the middle"?
A documented long-context failure mode where models recall information near the start and end of a long prompt reliably, but under-weight facts placed in the middle — even though the fact is technically present in the context. It means a bigger context window doesn’t guarantee even attention over everything in it.
Can you combine RAG and long context?
Yes — retrieve a broad candidate set (not just top-3) and let the model reason over all of it in one long-context call. This gets retrieval’s scalability over a large corpus and long context’s ability to connect facts across multiple retrieved documents at once.
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The RAG HandbookRetrieval-augmented generation end to end — why LLMs need it, chunking, embeddings and vector search, hybrid retrieval, two-stage reranking, grounded generation with citations, and how to evaluate a RAG system so a quietly-drifting index never fools you.Read →
- The Vector Databases HandbookHow vector search actually works — why brute-force kNN doesn't scale, distance metrics, approximate nearest neighbor, the HNSW, IVF and product-quantization indexes and their recall/latency/memory tradeoffs, metadata filtering, updates, scaling, and when you need a dedicated vector DB vs pgvector.Read →
- RAG vs Fine-TuningThe most common wrong turn in AI engineering — reaching for fine-tuning when you needed retrieval, or vice versa. The one distinction that decides it (knowledge vs behavior), a head-to-head table, when to use each, and why the strongest systems combine them: fine-tune for behavior, RAG for knowledge.Read →
- The Embeddings HandbookMeaning as geometry: how text and images become vectors so that similar meaning lands nearby, why cosine similarity measures direction not length, how L2-normalization makes the dot product equal cosine (so vector databases just take dot products), the curse of dimensionality, and how it all powers semantic search and RAG — with worked math and runnable code.Read →
- The Data Engineering for AI HandbookThe pipeline that turns raw documents into usable data for RAG and training — ingest, chunk, deduplicate, quality-filter. How overlapping chunking works, why deduplication restores diversity, and why the usable dataset is a compounding fraction of the raw pile (raw × dedup-rate × quality-rate) — because quality beats quantity for both retrieval and training. With worked yield math and a runnable pipeline.Read →
- Embeddings vs Keyword SearchEmbeddings match meaning even with zero shared words; keyword search (BM25) matches exact terms with a precision embeddings can’t guarantee. Why the two fail in genuinely complementary ways, and why hybrid search with fusion is the modern RAG default.Read →