The big idea
Why do agents need a memory system?
An LLM is stateless — it only knows what's in its context window right now. So a chatbot forgets your name the moment the conversation scrolls past the window, and remembers nothing from yesterday's session. For an agent or assistant meant to know you over weeks of conversations, that's fatal: no continuity, no personalization, no learning from past interactions.
An agent memory system gives a stateless model persistent memory: it decides what to remember from each conversation, stores it outside the context window, and recalls the relevant bits into the prompt when they matter — plus consolidates and forgets so memory stays coherent and bounded. It's RAG, but the corpus is the user's own history.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end wipe long-term memory and disable consolidation to see continuity and coherence break. Hit Begin.
Step 1 · Why not just keep everything?
Stuffing the history fails
The naive fix: append the entire conversation history to every prompt. It preserves everything — so why does it break down fast?
Design decision: Append the whole conversation history to every prompt. Why does this fail?
The call: History grows past the context window, costs more and slows every call, and still carries nothing across sessions. — Unbounded history eventually overflows any window, inflates token cost and latency on every turn, and dilutes the model's attention — and once a session ends, the window is gone, so there's no cross-session persistence at all. You need to store distilled memory outside the window and recall selectively.
Appending everything fails three ways: history eventually exceeds the context window; even before that it makes every call slower and costlier (and dilutes attention over long context); and it provides no cross-session persistence — when the session ends, the window is gone. The answer isn't a bigger window; it's to distill memory, store it outside the window, and recall selectively what's relevant now.
The window is not memory: The context window is working memory — small, transient, expensive per token. Real memory must live outside it, persistently, and be pulled in only when relevant. Conflating "put it in context" with "remember it" is the mistake; separating them is the whole design.
Step 2 · Two kinds of memory
Short-term vs long-term
Human memory isn't one thing, and neither is an agent's. What are the two layers, and what does each do?
Split memory in two. Short-term / working memory = the context window: the recent turns and the system prompt the model sees right now — small and transient. Long-term memory = a persistent store outside the window that survives across turns and sessions. The system's job is to move the salient parts of working memory into long-term storage (write), and pull the relevant parts of long-term storage back into working memory when needed (recall). Everything else is how you do those two moves well.
Working set + persistent store: It's the classic memory hierarchy: a tiny fast working set (context window) backed by a large persistent store (memory DB). You can't hold everything in the working set, so you page the relevant memories in on demand — exactly like caching, but for meaning.
Step 3 · What to remember
The write path
You can't store every word (that's the failed approach). So after a turn or session, you must decide what's worth keeping. How?
Design decision: You can't store everything. What do you write to long-term memory?
The call: The entire raw transcript, verbatim. — Storing raw transcripts recreates the unbounded-growth problem in the store and makes recall noisy. You distill: keep salient facts/events, not every word.
Run a memory writer that extracts the salient information — durable facts about the user, decisions, preferences, key events — usually with an LLM deciding what matters, and stores concise memories rather than the raw transcript. Decide when to write (after each turn, or a summary at session end) and what type (a fact vs an event). Distilling at write time keeps memory small, high-signal, and easy to recall — the opposite of dumping everything.
Distill, don't dump: The write path is where memory quality is set: an LLM extracting "user prefers Python, is building a startup, dislikes verbose answers" beats storing 50 messages. Good extraction (and typing memories as facts/events/preferences) makes everything downstream — recall, consolidation — work better.
Step 4 · Build the prompt
Assembling working memory
Every model call needs its context assembled from several sources under a hard token budget. What goes in, and how do you fit it?
A context builder assembles each prompt from: the system prompt, the recent turns (short-term), and the recalled long-term memories relevant to this turn — all packed within the token budget, prioritizing the most important/relevant and truncating or summarizing the rest. This is the moment memory becomes useful: the right past facts are placed into the window right when they're needed, so the stateless model behaves as if it remembers.
The context is a budget: Assembling context is a packing problem under a token limit: system + recent history + recalled memories, ranked by relevance/importance. Spend the budget on what helps this turn. The builder is the join point between working memory and long-term recall.
Step 5 · Where memories live
Vector + structured stores
Memories aren't all the same shape. "The user seemed frustrated last week" (fuzzy, recall by similarity) is different from "the user's name is Sam" (a fact you look up exactly). Where do they go?
Use the right store per memory type. A vector store holds embedded memories for semantic recall ("find memories related to what they're asking now") — great for episodic/fuzzy memory. A structured store / profile holds authoritative facts (name, preferences, settings) queried directly, not by similarity, so they're always correct and current. (Some systems add a knowledge graph for relationships.) Match storage to how each memory is written and recalled.
Semantic + structured: Episodic memory (events, gist) fits a vector store recalled by similarity; semantic facts about the user fit a structured profile queried exactly. Using one for the other hurts: you don't want the user's current name to depend on a fuzzy similarity match. Type your memory, store it accordingly.
Step 6 · Pull the right memories
The recall path
At each turn, of potentially thousands of stored memories, only a few are relevant. Injecting all of them is impossible (token budget) and injecting the wrong ones misleads the model. How do you recall well?
Design decision: Thousands of memories exist; only a few are relevant now. How do you recall the right ones?
The call: RAG over memory: embed the current turn, retrieve the most relevant memories (with recency/importance), inject those. — Treat the memory store as a corpus: embed the current query/turn, retrieve the top relevant memories by similarity (often blended with recency and an importance score), pull exact facts from the profile, and inject just those into the context. It's retrieval-augmented generation where the corpus is the user's own history.
Recall is RAG over memory. Embed the current turn, retrieve the most relevant memories from the vector store (often blending similarity with recency and an importance score), fetch exact facts from the profile, and inject just those into the context. Of thousands of memories, only the handful that matter now reach the model — enough for continuity, few enough to fit the budget and avoid misleading noise.
Relevance + recency + importance: Pure similarity isn't enough; good recall ranks by a mix of relevance to the current turn, how recent a memory is, and how important it was judged at write time. This surfaces the right past context without flooding the window — the read side of the memory system.
Step 7 · Keep it coherent
Consolidation & forgetting
Left alone, memory grows forever, accumulates duplicates, and holds contradictions (the user moved cities; both the old and new fact are stored). More memory then makes recall worse. How do you keep it healthy?
Run consolidation (a background process, like sleep for memory): summarize and merge related memories, deduplicate, resolve contradictions (newer/authoritative facts supersede older ones), and decay/forget stale or low-importance memories so the store stays bounded. This keeps recall precise and memory current — critically, an update to a fact (name change, new preference) must overwrite/supersede, not just add another contradictory memory.
Merge, resolve, decay: Without consolidation, memory bloats and contradicts itself, and similarity recall happily returns stale facts alongside current ones. Consolidation compresses (summaries of many episodes), reconciles conflicts, and forgets — so memory improves the agent instead of slowly poisoning it. Forgetting is a feature.
Step 8 · The sharp edges
Staleness, privacy & poisoning
Memory has dangerous failure modes: recalling stale/contradictory facts, poor recall precision (right memory not surfaced, or wrong one injected), sensitive PII stored indefinitely, and memory poisoning (a user planting false "facts" the agent later trusts).
Fight staleness with consolidation and by preferring the authoritative profile for facts (and timestamping memories). Improve recall precision with better ranking, importance scoring, and typed memories. Handle privacy: treat memory as sensitive user data — consent, encryption, retention limits, and a "forget me" that truly deletes. Defend against poisoning by not blindly trusting extracted "facts" (especially security-relevant ones), validating/attributing memories, and scoping memory per user. And watch cost — recall + write + consolidation are extra model/DB calls per interaction.
Design for the unhappy path: Stale facts → consolidate + prefer profile + timestamps. Bad recall → rank + type + score. PII → consent, encryption, deletion. Poisoning → don't blindly trust, attribute, scope per user. Memory makes an agent personal and powerful — and turns its data into a privacy and trust surface you must protect.
You did it
You just designed an agent memory system.
- LLMs are stateless — without memory they forget across the window and across sessions.
- Stuffing the whole history fails: it overflows the window, costs more, and never persists across sessions.
- Split memory: short-term (context window) vs long-term (persistent store), moving salient info between them.
- Write path: extract and store salient facts/events (an LLM decides), not the raw transcript.
- Assemble each prompt under a token budget: system + recent turns + recalled memories.
- Store by type (vector store for episodic/semantic recall, a structured profile for exact facts); recall is RAG over memory.
- Consolidate and forget — merge, dedup, resolve contradictions, decay — so memory stays coherent, current and bounded.
