A finite window,
an endless task.

Every LLM call is stateless — the model only knows what's in its context window right now. To give an agent continuity, you resend the conversation history each turn. But the window holds a fixed number of tokens, and a real agent's history — messages, tool calls, tool results, retrieved documents — grows without bound. The collision is inevitable: at some point the history no longer fits, and you must decide what to drop.

That decision is agent memory. It's not a fancy neural mechanism; it's a budgeting problem. You have a token budget (the window minus room for the model's reply and system prompt), a stream of content that exceeds it, and a policy for choosing what survives into the next turn and what gets stored elsewhere. Get the policy right and the agent stays coherent over hours; get it wrong and it forgets the thing it needed most, mid-task. Everything else in this handbook is refinements of that budget.

Memory splits into two tiers, mirroring how we usually think about it. Short-term (working) memory is the context window itself — the recent messages the model sees directly this turn. It's fast and fully available, but tiny and volatile: anything not in it, the model simply doesn't know. Long-term memory is an external store — a database or vector index — that holds facts, summaries, and past events indefinitely, outside the window.

The agent's job is to shuttle between them: keep the relevant recent turns in short-term memory, write durable facts to long-term memory, and retrieve from long-term back into the context when a turn needs them. This is exactly the architecture MemGPT framed as an operating system for memory — the context window is RAM, external memory is disk, and the agent pages information between them. With that split, an agent can "remember" far more than its window ever holds, because most of its memory lives on disk and only the relevant page is loaded at a time.

When the context exceeds budget, you have two moves to make room. Eviction drops the oldest messages until the rest fit — first-in-first-out, like a sliding window over the conversation. It's dead simple and keeps the most recent, most relevant turns, but it forgets: whatever falls off the back is gone from context entirely.

Summarization is the smarter move: replace a block of old messages with a short summary that preserves the essential facts for a fraction of the tokens. A rolling summary lets the agent carry a trace of the whole conversation in a small, fixed budget, so it doesn't fully forget the early context — it just holds a compressed version of it. The trade-off is fidelity: summaries lose detail, and a fact dropped from a summary is as gone as an evicted one. That's why the third leg — external memory you can retrieve exact facts from — matters, and why production agents combine all three.

The whole problem is a token inequality. The messages you send must fit the budget B (the window minus reply and system-prompt room). Eviction keeps the newest messages whose total stays under B:

FIFO: the newest turns survive, the oldest fall off. Simple, but everything evicted is gone from context.

Summarization changes the arithmetic. Replace a block of history costing T tokens with a summary costing a fraction ρ·T (the compression ratio, e.g. ρ=0.25), and you free most of that budget while keeping the gist:

Compressing old turns to ρ·T lets the agent retain a trace of history it would otherwise evict — trading detail for coverage. The runnable version below fits messages to a budget by eviction, compresses evicted history, and retrieves a fact from external memory.

Eviction and summarization both lose information — the only question is how gracefully. The fix for durable facts is an external memory store the agent can search. It writes facts and past events out (often to a vector index keyed by embeddings), and when a new turn needs something old, it retrieves the relevant items back into context — recalling a specific fact long after it scrolled out of the window.

The production recipe combines all of it: keep the last few turns verbatim (working memory), maintain a rolling summary of older turns (compression), and retrieve durable facts on demand from external memory (retrieval). This is the memory-stream idea behind generative agents — store observations, retrieve the relevant ones, and let the agent act with a memory far larger than its window. The window is just the desk; the filing cabinet is where most of the memory lives.

The subtle failure is summarizing away the load-bearing detail. A summary that reads well can quietly drop the exact number, id, or constraint the agent will need later — and once it's out of both context and summary, it's unrecoverable unless it also went to external memory. So route durable specifics (ids, decisions, user facts) to a store, and let summaries carry only the gist. Another trap is retrieval that fetches the wrong memories: bad relevance ranking pulls in irrelevant history and pushes out what mattered, so memory quality depends on retrieval quality just as much as RAG does.

Two more. Recall context rot — a bigger window is not a free pass, because models attend less reliably to the middle of a very long context, so stuffing everything in can hurt even when it fits. And beware unbounded external memory: a store that only ever grows makes retrieval slower and noisier over time, so agents need forgetting policies (decay, deduplication, importance weighting) for long-term memory too. Good agent memory isn't about remembering everything — it's about spending a small token budget on the right things, and knowing where the rest is filed.