Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

LABS · 19  /  CONTEXT

The Crowded Desk.

Give a model a bigger context window and it should get smarter, right? Bury one crucial fact — the needle — in a wall of filler and find out. Move it around the window, watch it sink in the middle, and watch recall rot as the desk fills. A fuller window is not a smarter one.

Act 1 · Position matters
Move the needle, watch recall shift
The window holds one gold needle fact, a blue question at the end, and a lot of gray filler. Slide the needle from top to bottom. Recall is strong at the edges and weakest in the middle — the U-shaped "lost in the middle" curve.
Needle positiontop
—
Recall probability
—
Where in window
—
Verdict
Same fact, same model — only the position changed, yet recall swings wildly. This is why prompt order is a design decision: put what matters near the top or, better, right next to the question.
Act 2 · The desk fills
Recall rots as you add filler
Now hold the needle in the middle and pour in filler. Attention is a finite budget spread over every token — the more low-signal content you pile on, the less reliably the model attends to the one fact that matters. Watch the curve fall.
Context fill8K tokens
—
Middle-needle recall
—
Prefill cost
relative to 8K
—
Signal : noise
Bigger window, worse recall AND higher cost — the double tax of context rot. A 128K prompt that’s 99% filler is slower, pricier, and less accurate than a tight 4K prompt with just the right facts.
Curate, don't cram
The fix isn't a bigger desk — it's a tidier one. Toggle the three moves of context engineering and watch recall recover: retrieve only what's relevant, reposition it next to the question, and compact the old filler away.
62%
Recall probability
128K
Tokens in window
1%
Signal density
Start with the rotted 128K window and add the moves. Each one raises recall and cuts tokens at once — because the goal was never "fit more", it was "the model reliably sees what matters".

Why more context can mean less intelligence

It feels like a bigger context window should strictly help — more room for more information. But two effects push the other way, and together they are context rot.

First, lost in the middle: models recall content at the start and end of a long context far better than content buried in the center. Retrieval accuracy traces a U by position — strong at the edges, weakest in the middle. Where a fact sits changes whether the model uses it at all.

Second, attention dilution: attention is a finite budget spread across every token in the window. Pour in filler and each token — including the one that matters — gets a thinner slice. Add the model's tendency to be distracted by irrelevant-but-plausible text, and accuracy sags as the window fills, regardless of position.

The counterintuitive law

A fuller window is not a smarter window. Bigger context raises the ceiling on what you can include and costs more prefill for the privilege — but the middle is still weakest and noise still dilutes. Capacity is not quality.

The discipline: context engineering

The answer is to curate the window rather than fill it — the whole point of context engineering. Retrieve only the relevant chunks instead of dumping everything. Position the most important content near the question, exploiting the strong edge rather than fighting the weak middle. Compact old turns into short summaries before they bury the signal. Keep durable facts in a small always-visible core instead of scrolling history.

Every one of those moves does the same thing: raises the signal density of the window. That — not window size — is what actually determines whether the model can find the needle.

The picture: a crammed window and a curated one

crammed: 128K instructions · edge filler needle · middle, weakest filler question · edge curated: 4K instructions pinned core facts summary of old turns retrieved chunks needle · beside question question retrievecompactreposition 200 relevant tokens: 0.16% of 128K, 2.5% of 8K signal density, not window size, decides recall
Act 1 and Act 3 of the lab in one frame. On the left the needle sits where recall is weakest, surrounded by tokens that each take a share of attention. On the right nothing new was added. Filler was removed (retrieve), history was shrunk (compact), and the one fact that matters was moved to the strong edge, next to the question (reposition). The window got smaller and the needle got easier to find, for the same reason.

Trade-offs: what each fix costs

Here Time means latency and extra calls, Space means tokens and storage, and Why is what you risk. None of these is free. The point is to pay where it buys recall.

CaseTimeSpaceWhy
Bigger window, same promptPrefill grows with every token; 8K to 128K is 16× the tokens and up to 256× the pairwise attention work16× the input tokens billedRaises the ceiling on what fits. It does nothing for the weak middle, and adds more distractors.
Retrieve top-k chunksAn index lookup (and often a reranker) before every callA vector or keyword index to build and keep freshUsually the biggest single win. Recall now depends on the retriever: a chunk it misses is a chunk the model never sees.
Reposition key factsFreeNoneUses the strong edges instead of fighting the middle. You have to know which facts are key, which is what retrieval scores tell you.
State the task first, ask the question lastFreeA few dozen tokens for the repeatPuts the instructions and the question at the two strong edges, with the documents between them. It is a cheap first step before anything in the rows below.
Compact old turnsAn extra model call to summariseThe summary’s tokens, on every later callLossy. A detail the summary dropped is gone for good, so keep exact values (ids, numbers, decisions) out of the summary and in pinned facts.
Pinned core factsNone per callA small fixed token cost on every callDurable facts stay visible at an edge. Stale pins are worse than none, so they need updating.
Prompt caching of a stable prefixFaster prefill on repeat callsCache reads are billed at a discount; the details differ by providerCuts cost and latency only. The cached tokens are still in the window, so recall is unchanged.
Split into several calls, then combinek calls instead of one, plus a combine stepEach window stays smallEvery call sees a dense window, but no single call sees two facts that sit in different chunks.

Read down the Why column and two kinds of fix stand out. Bigger windows and caching change what a call costs. Retrieval, repositioning and compaction change what the model actually sees, and those are the only ones in the table that move recall. The lab’s own end states show the size of the change. Act 3 goes from 128K tokens to 4K: 32 times fewer tokens to read and pay for, and the same 200 relevant tokens go from 0.16% of the window to 5%. You cut the bill and raise the signal with one change.

Where this shows up in production and interviews

  • Where the U comes from

    The position effect in Act 1 was measured in the paper “Lost in the Middle: How Language Models Use Long Contexts” (Liu et al., 2023). On multi-document question answering and key-value retrieval, accuracy was highest when the relevant item was near the start or end of the input and dropped when it sat in the middle. Long-context behaviour has changed since then and varies by model, which is why the page says to re-test on every model change.

  • RAG pipelines

    A retrieval-augmented app is context engineering in code. It retrieves a few chunks, often reranks them, and decides their order in the prompt. Pipelines that dump the top 50 chunks “just in case” reproduce the left-hand window above, at 50 chunks’ cost.

  • Long-running agents and chat

    Coding agents and chat assistants fill their window with tool output and history over a session. They summarise older turns once the window gets close to full (Claude Code’s /compact command does this on request) and keep project facts in a file that is loaded on every turn. That file is the pinned core.

  • The interview version

    “The right document is in the context, but the bot still answers wrong. Why?” Work through it: where in the window the passage sits, how many near-duplicate distractors surround it, how long the prompt is, and whether the question asks for the fact explicitly. Then propose fixes in cost order: reorder, trim, retrieve fewer and better chunks, and evaluate with realistic distractors.

Context rot, step by step

A toy model of why a crowded window hurts, the arithmetic of what you pay for it, and a checklist for keeping signal density high. The widget's recall curves are illustrative; the numbers below are exact for the toy model they describe, not a benchmark of any real model.

How a crowded window dilutes attention

  1. Score. Each query scores every token in the window. Give the needle a score D higher than each distractor.
  2. Normalize. Softmax splits one unit of attention across all tokens: the needle gets e^D / (e^D + n − 1) when n − 1 distractors tie.
  3. Watch n grow. The numerator stays fixed while the denominator grows with n, so the needle's share falls as the window fills unless its score gap grows too.
  4. Add position effects. Real models are also unevenly sensitive to where a token sits, which is the U-shaped recall in Act 1.

Worked example

  1. A gap of 4 (a needle roughly 55 times as attractive as each distractor): at 1,000 tokens it receives 5.2% of attention, at 10,000 tokens 0.54%, at 100,000 tokens 0.05%.
  2. A gap of 6 (about 400 times): 28.8%, 3.9% and 0.40% at the same sizes. Ten times the filler removes about 90% of the share in either case.
  3. Why models still work. Real attention is spread over many heads and layers and sharpens when a query matches strongly, so the loss is far smaller than this one-head toy suggests. The direction is the same, and the toy shows why distractors that resemble the needle hurt most: they shrink the gap D.
  4. Cost. Going from 8K to 128K tokens is 16 times the tokens to read and, for the attention step alone, up to 256 times the pairwise work. Providers also bill per input token.
  5. Signal density. 200 relevant tokens are 2.5% of an 8K window but 0.16% of a 128K one.

Knobs and pitfalls

Retrieve a few high-precision chunks instead of whole documents; put instructions and key facts at the start and the question at the end; summarize old turns and keep a small pinned core of facts. Evaluate with realistic distractors and several needles, not a single easy needle in blank text, because passing that test does not show the model uses a full window well. Re-test whenever you change models, since long-context behavior varies a great deal between them.

Check yourself

Four questions. Don't let them rot.

See every answer, with the reasoning
  1. What is the "lost in the middle" effect?

    Answer: Models recall content at the start and end of a long context better than the middle — Retrieval accuracy is U-shaped by position — edges strong, middle weakest.

  2. Does a bigger context window fix context rot?

    Answer: No — the middle is still weakest, noise still dilutes attention, and cost rises — Bigger raises the ceiling and the bill, not the quality. Capacity ≠ quality.

  3. Why does adding filler hurt recall of an unchanged fact?

    Answer: Attention is a finite budget spread over all tokens — more filler, thinner slice for the needle — Dilution: every extra low-signal token steals attention from the one that matters.

  4. What is the core move of context engineering against rot?

    Answer: Raise the signal density — retrieve the relevant, position it well, compact the rest — Curate, don’t cram. Signal density, not window size, decides whether the needle is found.

Finished this one? 0 / 61 Labs done
NextThe Loop: Agent Loop Simulator Closest lab to this one

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs