AI~9 min readComparison
Head to Head
RAG vs long context: retrieve, or paste it all in?
RAGvsLong Context
A 1M-token context window makes it tempting to skip retrieval entirely and just paste your whole knowledge base into the prompt. Sometimes that's the right call. But bigger context doesn't mean the model reads it evenly, and it doesn't mean it's cheap — which is exactly what this comparison is about.
01
The one distinction that decides everything
RAG selects a small, relevant slice of a large corpus at query time via search, and puts only that slice in the prompt. Long context skips selection entirely — you paste a large chunk of (or the entire) corpus directly into the prompt and let the model's attention mechanism find what matters on its own.
→ The rule
If your corpus is bigger than fits comfortably in a context window — or changes often enough that re-sending it every call is wasteful — you need retrieval. If your corpus is fixed and modest, and the task genuinely requires reasoning ACROSS the whole thing at once, long context wins outright.
| Dimension | RAG | Long Context |
|---|
| Corpus size | Effectively unbounded (index scales) | Bounded by the context window, however large |
| Cost per query | Low — small retrieved slice + short prompt | High — pays for prefill over every token sent, every call |
| Freshness | Instant — update the index | Must re-send (and often re-cache) the whole corpus |
| Precision on a needle | High if retrieval finds the right chunk | Degrades — "lost in the middle" attention dilution |
| Cross-document reasoning | Limited to top-k retrieved chunks | Strong — model sees everything jointly, at once |
| Citations | Natural — cite the retrieved chunk | Possible but not built-in, needs extra prompting |
| Setup complexity | Chunking + embedding + index pipeline | None — paste text in |
| Failure mode | Bad retrieval = model never sees the answer | Dilution = model sees it but under-weights it |
Reach for RAG
- Corpus is large, unbounded, or grows continuously
- Content updates often — docs, tickets, prices
- Cost per query matters at real request volume
- You need citations back to a specific source
- Most of the corpus is irrelevant to any given query
Reach for long context
- Corpus is fixed-size and fits the budget (a contract set, a codebase)
- The task needs holistic reasoning across everything at once
- You want zero retrieval infrastructure
- The same context is reused across many calls (prompt caching amortizes it)
- Few-shot examples or a whole reference doc, not a huge KB
04
The answer is usually "retrieve, then go long"
The strongest production pattern isn't RAG or long context — it's RAG feeding a long context window. Instead of retrieving only the top 3 chunks (which can lose cross-document connections), retrieve a broader top-50 or top-100 candidate set and let the model reason over all of them jointly in one long-context call. Retrieval narrows an unbounded corpus down to a size that fits; long context then lets the model actually connect the pieces instead of reasoning over 3 isolated fragments.
→ The cheap default
Start with RAG at a small top-k — it's the cheapest, fastest option and solves most lookup-shaped questions. Widen the retrieved set and lean on long-context reasoning only when you've seen retrieval miss cross-document connections a small top-k can't hold at once.
05
Why bigger context doesn’t mean evenly-read context
It's tempting to assume a model with a 1M-token window "reads" all 1M tokens the same way it reads a short prompt — but empirically, that's not how attention behaves. Research on long-context recall (the "lost in the middle" finding) shows models are reliably strong at attending to information near the start and end of a context, and measurably weaker at recalling a fact buried in the middle of a long document, even though nothing was hidden and the token was technically "in the prompt." The context window is a hard ceiling on what the model CAN see; it says nothing about how evenly the model actually weighs what it sees.
This is structurally different from RAG's failure mode. When RAG fails, it's usually because the retriever never fetched the right chunk — a visible, debuggable failure you can fix by tuning the retriever. When long-context recall fails, the right fact WAS in the prompt the whole time, and the model simply under-weighted it relative to information sitting nearer the edges — a much quieter, harder-to-detect failure, because from the outside a plausible-but-wrong answer looks identical to a well-reasoned one.
→ The trade you’re actually making
RAG trades "did I retrieve the right thing" (visible, tunable) for long context's "did the model actually weigh what I gave it evenly" (invisible until you specifically test for it with needle-in-a-haystack evals).
06
A worked scenario: legal contract review vs a support knowledge base
A tool reviewing a single 200-page merger contract for conflicting clauses is a long-context problem: the corpus is fixed, modest, and the whole point is cross-clause reasoning — clause 14 might contradict clause 187, and RAG's top-k retrieval could easily never place those two chunks in the same call together. Pasting the whole contract into a long-context window (with prompt caching amortizing the cost across the many questions asked about that one contract) is both simpler and more correct here.
A customer-support bot backed by a million-document, hourly-updated knowledge base is the opposite case: the corpus vastly exceeds any context window, most of it is irrelevant to any single question, and updates need to be live within minutes, not require re-sending gigabytes of docs. RAG's index-and-retrieve model is the only one of the two that scales here — and if a question genuinely needs to synthesize across a handful of related articles, that's exactly the "retrieve-broad-then-go-long" combination from section 04, not a reason to abandon retrieval.
→ The pattern generalizes
Fixed + small + needs holistic reasoning → long context. Unbounded + changing + mostly-irrelevant-per-query → RAG. Most real systems have both shapes of problem inside them, at different points.
Frequently asked
Quick answers
RAG or long context — which should I use?
Decide by corpus shape. If it’s large, unbounded, or updates often, use RAG. If it’s fixed-size, fits the budget, and the task needs holistic reasoning across the whole thing, use long context. Many production systems retrieve a broad candidate set and then reason over it in a long-context call.
Does a bigger context window make RAG unnecessary?
No. A bigger window raises the ceiling on how much you CAN paste in, but it doesn’t fix "lost in the middle" attention dilution, and it doesn’t fix the cost of re-sending a huge prompt on every single query. For unbounded, frequently-updated corpora, RAG stays necessary regardless of window size.
What is "lost in the middle"?
A documented long-context failure mode where models recall information near the start and end of a long prompt reliably, but under-weight facts placed in the middle — even though the fact is technically present in the context. It means a bigger context window doesn’t guarantee even attention over everything in it.
Can you combine RAG and long context?
Yes — retrieve a broad candidate set (not just top-3) and let the model reason over all of it in one long-context call. This gets retrieval’s scalability over a large corpus and long context’s ability to connect facts across multiple retrieved documents at once.