Gemini 1.5,
explained.

Rungs 1 and 2 scale parameters and compute. This rung scales the context window, by roughly 5 to 80 times over what was then normal — from 128 to 200 thousand tokens to one million, with ten million demonstrated — in a sparse mixture-of-experts model.

The claim that made it credible is a measurement rather than a capacity, and the distinction is the whole point. A long window is easy to advertise and easy to fake: a model can accept a million tokens and attend to almost none of them usefully. The test reported is needle-in-a-haystack — hide a specific fact at an arbitrary position in the full context and ask for it back — and the reported recall is near-perfect across the whole window, not merely near its ends.

Stated against the context ladder, that result is a direct refutation of a family of approaches. A model whose attention is restricted to a neighbourhood cannot pass this test at position half-a-million, because no attention operation ever connects the needle to the query — the information has to be relayed through depth, and it is not. Passing it everywhere is evidence of something close to genuine global access, which is why this rung is consistent with keeping attention exact and sharding it rather than approximating it — though the report does not disclose how Gemini achieves it.

Every earlier leap in language models was about the model. Gemini 1.5's leap is about the prompt: how much you can put in front of the model at once. Its context window reaches 1 million tokens in production and up to 10 million in research — enough to hold entire books, whole codebases, or hours of audio and video in a single request. That changes what a model can do: instead of retrieving snippets and hoping, you can hand it the whole document and ask.

Architecturally Gemini 1.5 is a sparse mixture-of-experts transformer — only a fraction of its parameters fire for any given token (which cuts per-token compute, though not the cost of attending over a long context). But a huge window is only useful if the model can actually use all of it. A prompt box that holds a million tokens but only pays attention to the last few thousand is a bigger box, not a longer memory. The proof that Gemini 1.5's memory is real is a deceptively simple test.

The needle-in-a-haystack test makes "does it really use the whole context?" measurable. Take a huge block of filler text (the haystack), hide one small, specific fact inside it (the needle — say, a random passkey), then ask the model to retrieve just that fact. Sweep the needle across many depths (start, middle, end) and many context lengths, and you get a grid of recall scores: green where the model found it, red where it didn't.

For most long-context models that grid lights up red in the early depths of long contexts — the needle placed near the beginning of a very long document is lost. Gemini 1.5's grid was near-uniformly green, ~99%+ recall across the whole range up to a million tokens. Depth barely mattered; length barely mattered. That uniformity is the claim: not "a big window" but "a window it can search end to end."

To see why this is hard, imagine a model whose effective attention reaches only the last W tokens — a sliding window. Anything older than W tokens has scrolled out of view. Put the needle near the end of the context and it's inside the window: found. Put it near the start of a long context and it's beyond W: gone, no matter how big the prompt box is.

So the interesting number isn't the size of the prompt box — it's the usable span. A model advertising a million-token window but effectively attending to only the last 8K would ace needles at the end and fail everything earlier. Gemini 1.5's achievement is that its usable span really does stretch across the entire window, which is exactly the condition the next section makes precise.

Place a needle at depth-fraction f (0 = start, 1 = end) in a context of length L. Its index is about f·(L−1), so its distance from the end is:

A needle at the very end (f=1) is distance 0; at the very start (f=0) it is nearly the whole length away.

A window model attending to the last W tokens retrieves the needle exactly when that distance fits inside the window:

To guarantee recall at every depth, including the very start (f=0), the span must cover the whole context: W ≥ L. That single inequality is the whole long-context game — and it comes at a price, because both self-attention (naively quadratic in L) and the KV cache that stores past tokens (linear in L, ~tens of GB at a million tokens) grow with the length you're trying to span. Gemini 1.5's undisclosed architecture changes and serving optimizations are what make W ≥ L ≈ 10⁶ affordable; MoE alone does not. The runnable version below builds the recall grid and shows a small window collapsing while a full-span one stays perfect.

The report describes the model only in outline and its long-context design not at all, so the substance is in how the context was tested, from synthetic probes to realistic tasks.

1. Start from what is disclosed. Gemini 1.5 Pro is a sparse mixture-of-experts Transformer: a learned router sends each input to a subset of the parameters, so total size can grow while the number active per input stays fixed. The report credits the long context to significant architecture changes it does not describe. The models were trained on multiple 4,096-chip pods of TPUv4 across data centers, on multimodal and multilingual data, then instruction-tuned and tuned on human preferences. The updated report adds Gemini 1.5 Flash, distilled from Pro, which computes attention and feed-forward layers in parallel, as PaLM did.
2. Measure prediction by position. On long documents of up to 1 million tokens, and on code repositories built by shuffling and concatenating files up to 10 million, the report plots the cumulative average negative log-likelihood up to each token position and fits a power law. If far-back tokens help, the curve keeps falling.
3. Hide a needle in text. The haystack is Paul Graham essays, concatenated and repeated to the target length. The needle is a sentence giving a special magic number for a named city, inserted at evenly spaced depths from the start to the end, and the model is asked for that city's number. The grid sweeps lengths up to 10 million tokens.
4. Hide needles in video and audio. For video, a line saying the secret word is needle is overlaid on one random frame of a 10.5-hour video made from seven copies of the AlphaGo documentary, sampled at one frame per second: 9.9 million tokens. For audio, a few seconds of a speaker saying the secret keyword is hidden in up to 107 hours of multi-speaker speech from VoxPopuli. The question is always asked in text.
5. Ask for many needles at once. A harder variant hides 100 distinct needles and asks for all of them in one turn, which tests whether retrieval holds up when it is not a single lookup.
6. Move to realistic tasks. Long-document and long-video question answering, long-context speech recognition, and learning a language from its documentation: for Machine Translation from One Book, the field linguistics material for Kalamang goes into the prompt: a grammar of roughly 500 pages, a word list of about 2,000 entries and around 400 parallel sentences, about 250,000 tokens in all.
7. Control for prior knowledge. Kalamang has fewer than 200 speakers and almost no presence on the web, and every model is also run 0-shot with no materials. Because GPT-4 Turbo (128K) and Claude 3 (200K) cannot hold the full materials, all models are also compared on half the grammar, about 100,000 tokens. The reference is a person who learned Kalamang from the same materials, who also rates the translations from 0 to 6.

• x — token position; NLLavg(x) averages the loss over the first x tokens of a document or repository.
• wt — the token at position t, predicted from everything before it.
• α, β — the fitted scale and exponent; γ — the level the curve approaches.

The curve is a test of whether distant context is used. A model that gains nothing from history beyond W tokens stops improving at position W, so its cumulative average levels off; the report shows Gemini 1.0 Pro improving only up to 32K tokens. Gemini 1.5 Pro's average kept falling to 1 million tokens on documents and 10 million on code. The power law fit well to about 1 million tokens for documents and about 2 million for code; near 10 million the code curve fell below the fit, which the report ties to repeated code blocks giving the model an outsized benefit and flags as possibly specific to the dataset. Read at the extremes, β near 0 would mean extra context buys nothing, and a curve that beats its fit can mean copying rather than understanding.

The figures are from the updated report, which also gives the newer Gemini 1.5 Flash near-perfect recall up to 2 million tokens in all three modalities.

Several results mark the limits. Recall of 100 needles at once is far below single-needle recall, so finding one fact is not the same as gathering many, and GPT-4 Turbo's recall on that task oscillated as context grew. On the Kalamang benchmark every model, Gemini included, was essentially random with no materials in context, which confirms the language is not in the training data. With the materials, Gemini 1.5 Pro translated English to Kalamang about as well as the human learner on average, while Kalamang to English still trailed the learner noticeably; with half the grammar, it beat GPT-4 Turbo and Claude 3 on the same setup by a wide margin. The report also states that Gemini 1.5 Pro matches or beats Gemini 1.0 Ultra on a broad set of benchmarks while using significantly less training compute. All of these are evaluations run by the vendor on haystacks it built.

The limitation is in what the benchmark rewards, and the field took several years to internalise it. Needle-in-a-haystack tests exact retrieval of a verbatim span. That is the single easiest thing to do with a long context and the least representative of why anyone wants one. The valuable cases are aggregation — summarise this whole book, find every place these two claims conflict, reason over facts scattered across five chapters — and a model can score perfectly on retrieval while degrading badly on all of them. The top of the context ladder is an argument about exactly this gap.

The cost side is also easy to skip past: a million-token prompt is a million-token prompt in compute and in price, every time it is sent, and the practical question of when that beats retrieving the relevant ten thousand tokens is an economic one the capability does not answer. Figures here are vendor-reported.

How it held up: very long context became a standard frontier feature rather than a differentiator, and needle-in-a-haystack became a standard benchmark and then, fairly quickly, a saturated one — which is the normal life cycle of a test that a specific architectural choice makes easy. The durable contribution is that it made "does this window actually work" a question with an answer, and the durable caution is that its answer was narrower than the headline.

Gemini 1.5 reframed a running debate: do you retrieve relevant chunks and feed a short context (RAG), or do you just put everything in a long one? A million-token window doesn't end retrieval — retrieval is still cheaper and scales past any fixed window — but it dramatically raises the bar for when you need it. Many tasks that used to demand a retrieval pipeline become "paste the whole thing and ask." The engineering shifts from chunking and indexing to managing one very large, very expensive context.

It also made long-context evaluation serious. Needle-in-a-haystack is only the floor — it tests retrieval, not reasoning across a long context — and the field quickly moved to harder tests of multi-fact synthesis over long inputs. But the simple passkey grid remains the honest first question to ask any long-context model: not "how big is your window?" but "can you actually find something at the far end of it?" Gemini 1.5 was the model that made the answer, convincingly, yes.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Four papers that are reports, not methods: they state capability and withhold recipe. Read for the evaluation design and the omissions, not the architecture.

1. 1PaLM2022
2. 2GPT-4 Technical Report2023
3. 3Gemini 1.52024
4. 4GPT-5.6 System Card2026

All 110 papers, in 19 ladders →