RAG vs long context: retrieve, or paste it all in?

RAG selects a small, relevant slice of a large corpus at query time via search, and puts only that slice in the prompt. Long context skips selection entirely — you paste a large chunk of (or the entire) corpus directly into the prompt and let the model's attention mechanism find what matters on its own.

The strongest production pattern isn't RAG or long context — it's RAG feeding a long context window. Instead of retrieving only the top 3 chunks (which can lose cross-document connections), retrieve a broader top-50 or top-100 candidate set and let the model reason over all of them jointly in one long-context call. Retrieval narrows an unbounded corpus down to a size that fits; long context then lets the model actually connect the pieces instead of reasoning over 3 isolated fragments.

It's tempting to assume a model with a 1M-token window "reads" all 1M tokens the same way it reads a short prompt — but empirically, that's not how attention behaves. Research on long-context recall (the "lost in the middle" finding) shows models are reliably strong at attending to information near the start and end of a context, and measurably weaker at recalling a fact buried in the middle of a long document, even though nothing was hidden and the token was technically "in the prompt." The context window is a hard ceiling on what the model CAN see; it says nothing about how evenly the model actually weighs what it sees.

This is structurally different from RAG's failure mode. When RAG fails, it's usually because the retriever never fetched the right chunk — a visible, debuggable failure you can fix by tuning the retriever. When long-context recall fails, the right fact WAS in the prompt the whole time, and the model simply under-weighted it relative to information sitting nearer the edges — a much quieter, harder-to-detect failure, because from the outside a plausible-but-wrong answer looks identical to a well-reasoned one.

A tool reviewing a single 200-page merger contract for conflicting clauses is a long-context problem: the corpus is fixed, modest, and the whole point is cross-clause reasoning — clause 14 might contradict clause 187, and RAG's top-k retrieval could easily never place those two chunks in the same call together. Pasting the whole contract into a long-context window (with prompt caching amortizing the cost across the many questions asked about that one contract) is both simpler and more correct here.

A customer-support bot backed by a million-document, hourly-updated knowledge base is the opposite case: the corpus vastly exceeds any context window, most of it is irrelevant to any single question, and updates need to be live within minutes, not require re-sending gigabytes of docs. RAG's index-and-retrieve model is the only one of the two that scales here — and if a question genuinely needs to synthesize across a handful of related articles, that's exactly the "retrieve-broad-then-go-long" combination from section 04, not a reason to abandon retrieval.

Long context re-sends the whole document set every call, so cost scales with total corpus size D; RAG retrieves only the slice a query actually needs, k tokens, so it sends roughly k/D of the tokens per query.