Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

LABS · 19  /  CONTEXT

The Crowded Desk.

Give a model a bigger context window and it should get smarter, right? Bury one crucial fact — the needle — in a wall of filler and find out. Move it around the window, watch it sink in the middle, and watch recall rot as the desk fills. A fuller window is not a smarter one.

Act 1 · Position matters
Move the needle, watch recall shift
The window holds one gold needle fact, a blue question at the end, and a lot of gray filler. Slide the needle from top to bottom. Recall is strong at the edges and weakest in the middle — the U-shaped "lost in the middle" curve.
Needle positiontop
—
Recall probability
—
Where in window
—
Verdict
Same fact, same model — only the position changed, yet recall swings wildly. This is why prompt order is a design decision: put what matters near the top or, better, right next to the question.
Act 2 · The desk fills
Recall rots as you add filler
Now hold the needle in the middle and pour in filler. Attention is a finite budget spread over every token — the more low-signal content you pile on, the less reliably the model attends to the one fact that matters. Watch the curve fall.
Context fill8K tokens
—
Middle-needle recall
—
Prefill cost
relative to 8K
—
Signal : noise
Bigger window, worse recall AND higher cost — the double tax of context rot. A 128K prompt that’s 99% filler is slower, pricier, and less accurate than a tight 4K prompt with just the right facts.
Act 3 · Context engineering
Curate, don't cram
The fix isn't a bigger desk — it's a tidier one. Toggle the three moves of context engineering and watch recall recover: retrieve only what's relevant, reposition it next to the question, and compact the old filler away.
62%
Recall probability
128K
Tokens in window
1%
Signal density
Start with the rotted 128K window and add the moves. Each one raises recall and cuts tokens at once — because the goal was never "fit more", it was "the model reliably sees what matters".

Why more context can mean less intelligence

It feels like a bigger context window should strictly help — more room for more information. But two effects push the other way, and together they are context rot.

First, lost in the middle: models recall content at the start and end of a long context far better than content buried in the center. Retrieval accuracy traces a U by position — strong at the edges, weakest in the middle. Where a fact sits changes whether the model uses it at all.

Second, attention dilution: attention is a finite budget spread across every token in the window. Pour in filler and each token — including the one that matters — gets a thinner slice. Add the model's tendency to be distracted by irrelevant-but-plausible text, and accuracy sags as the window fills, regardless of position.

The counterintuitive law

A fuller window is not a smarter window. Bigger context raises the ceiling on what you can include and costs more prefill for the privilege — but the middle is still weakest and noise still dilutes. Capacity is not quality.

The discipline: context engineering

The answer is to curate the window rather than fill it — the whole point of context engineering. Retrieve only the relevant chunks instead of dumping everything. Position the most important content near the question, exploiting the strong edge rather than fighting the weak middle. Compact old turns into short summaries before they bury the signal. Keep durable facts in a small always-visible core instead of scrolling history.

Every one of those moves does the same thing: raises the signal density of the window. That — not window size — is what actually determines whether the model can find the needle.

Context rot, step by step

A toy model of why a crowded window hurts, the arithmetic of what you pay for it, and a checklist for keeping signal density high. The widget's recall curves are illustrative; the numbers below are exact for the toy model they describe, not a benchmark of any real model.

How a crowded window dilutes attention

  1. Score. Each query scores every token in the window. Give the needle a score D higher than each distractor.
  2. Normalize. Softmax splits one unit of attention across all tokens: the needle gets e^D / (e^D + n − 1) when n − 1 distractors tie.
  3. Watch n grow. The numerator stays fixed while the denominator grows with n, so the needle's share falls as the window fills unless its score gap grows too.
  4. Add position effects. Real models are also unevenly sensitive to where a token sits, which is the U-shaped recall in Act 1.

Worked example

  1. A gap of 4 (a needle roughly 55 times as attractive as each distractor): at 1,000 tokens it receives 5.2% of attention, at 10,000 tokens 0.54%, at 100,000 tokens 0.05%.
  2. A gap of 6 (about 400 times): 28.8%, 3.9% and 0.40% at the same sizes. Ten times the filler removes about 90% of the share in either case.
  3. Why models still work. Real attention is spread over many heads and layers and sharpens when a query matches strongly, so the loss is far smaller than this one-head toy suggests. The direction is the same, and the toy shows why distractors that resemble the needle hurt most: they shrink the gap D.
  4. Cost. Going from 8K to 128K tokens is 16 times the tokens to read and, for the attention step alone, up to 256 times the pairwise work. Providers also bill per input token.
  5. Signal density. 200 relevant tokens are 2.5% of an 8K window but 0.16% of a 128K one.

Knobs and pitfalls

Retrieve a few high-precision chunks instead of whole documents; put instructions and key facts at the start and the question at the end; summarize old turns and keep a small pinned core of facts. Evaluate with realistic distractors and several needles, not a single easy needle in blank text, because passing that test does not show the model uses a full window well. Re-test whenever you change models, since long-context behavior varies a great deal between them.

Check yourself

Four questions. Don't let them rot.

See every answer, with the reasoning

  1. What is the "lost in the middle" effect?

    Answer: Models recall content at the start and end of a long context better than the middle — Retrieval accuracy is U-shaped by position — edges strong, middle weakest.

  2. Does a bigger context window fix context rot?

    Answer: No — the middle is still weakest, noise still dilutes attention, and cost rises — Bigger raises the ceiling and the bill, not the quality. Capacity ≠ quality.

  3. Why does adding filler hurt recall of an unchanged fact?

    Answer: Attention is a finite budget spread over all tokens — more filler, thinner slice for the needle — Dilution: every extra low-signal token steals attention from the one that matters.

  4. What is the core move of context engineering against rot?

    Answer: Raise the signal density — retrieve the relevant, position it well, compact the rest — Curate, don’t cram. Signal density, not window size, decides whether the needle is found.

Keep going

Handbook

Context Engineering →
Path

Master Context Engineering →
Tool

Context Window Visualizer →
Paper

MemGPT →
Finished this one? 0 / 61 Labs done
NextThe Loop: Agent Loop Simulator Closest lab to this one

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

AI Agents & Tools

More Labs

Explore more

Challenges →