The whole design, in writing
Learn AI system design by building a long-term memory system for an LLM agent or chatbot step by step. An interactive guide covering why stuffing the whole history into context fails, short-term vs long-term memory, deciding what to remember, assembling working context under a token budget, storing memories in vector + structured stores, recalling relevant memories, consolidation and forgetting, and the unhappy paths (staleness, contradictions, privacy, poisoning).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why do agents need a memory system?
An LLM is stateless — it only knows what's in its context window right now. So a chatbot forgets your name the moment the conversation scrolls past the window, and remembers nothing from yesterday's session. For an agent or assistant meant to know you over weeks of conversations, that's fatal: no continuity, no personalization, no learning from past interactions.
An agent memory system gives a stateless model persistent memory: it decides what to remember from each conversation, stores it outside the context window, and recalls the relevant bits into the prompt when they matter — plus consolidates and forgets so memory stays coherent and bounded. It's RAG, but the corpus is the user's own history.
What the new pieces do
- Userclient
- Talks to the agent across many turns and many sessions over weeks — and expects it to remember who they are and what they said.
Step 1 · Why not just keep everything?
Stuffing the history fails
The naive fix: append the entire conversation history to every prompt. It preserves everything — so why does it break down fast?
Append the whole conversation history to every prompt. Why does this fail?
Not punctuation. The history grows unbounded, so it eventually exceeds the context window, and even before that it makes every call slower and more expensive — and it still can't carry anything across separate sessions.
A bigger window only delays the wall and makes every call costlier and slower; history still grows without bound and long context degrades attention. And a window, however large, doesn't persist across sessions at all.
Unbounded history eventually overflows any window, inflates token cost and latency on every turn, and dilutes the model's attention — and once a session ends, the window is gone, so there's no cross-session persistence at all. You need to store distilled memory outside the window and recall selectively.
Appending everything fails three ways: history eventually exceeds the context window; even before that it makes every call slower and costlier (and dilutes attention over long context); and it provides no cross-session persistence — when the session ends, the window is gone. The answer isn't a bigger window; it's to distill memory, store it outside the window, and recall selectively what's relevant now.
Why this piece earns its place
The version of this that actually ships is rarely the naive one — it is a rolling window that drops the oldest turns, and that is where the quiet bug lives. Truncation is a correctness decision wearing the costume of a buffer policy: the turn you evict is often the one that set a constraint ("never email the client directly"), and nothing errors when it falls off the front. The agent simply stops honoring an instruction it was given forty turns ago, while the transcript the user is looking at still contains it — so the bug reproduces for nobody. There is a cost detail worth knowing too. An append-only prefix is the one shape prompt caching rewards, because the leading tokens are byte-identical turn after turn. The moment you summarize or reorder anything near the front, every cached prefix is invalidated and each turn pays full price again, so a naive trim can push the per-call bill up. And skip all of this when sessions are genuinely short and one-off: a rolling window is the right answer for a tool nobody comes back to.
Step 2 · Two kinds of memory
Short-term vs long-term
Human memory isn't one thing, and neither is an agent's. What are the two layers, and what does each do?
Split memory in two. Short-term / working memory = the context window: the recent turns and the system prompt the model sees right now — small and transient. Long-term memory = a persistent store outside the window that survives across turns and sessions. The system's job is to move the salient parts of working memory into long-term storage (write), and pull the relevant parts of long-term storage back into working memory when needed (recall). Everything else is how you do those two moves well.
Why this piece earns its place
Drawing the line in two immediately raises the first real operational question: when does something said this turn become recallable? If the writer runs at session end, and the builder only ever sees long-term memory plus the last few turns, then anything stated early in a long session is invisible to recall for the rest of it — the agent contradicts something the user told it twenty minutes ago while that line is still on screen. Short-term has to cover that gap, which means the window is not only a cost constraint, it is the read-your-own-writes layer. The split also draws a line between two blast radii, and that is the half people skip. A window belongs to one conversation and dies with it, so a turn that went badly is escaped by opening a new chat. The store belongs to the user: it is shared by every session they have open — the phone, the laptop, a job running in the background — and it follows them into every conversation they start after this one. Two of those sessions can be writing into it in the same minute, neither aware the other exists, and nobody restarts their way out of whatever lands there.
- short-termcontext window (transient)
- long-termpersistent store
- movewrite ↔ recall
What the new pieces do
- Agentgateway
- The stateless LLM app. Each call it must be handed the right context, and after each turn decide what is worth remembering.
Step 3 · What to remember
The write path
You can't store every word (that's the failed approach). So after a turn or session, you must decide what's worth keeping. How?
You can't store everything. What do you write to long-term memory?
A memory-writing step (often an LLM) reads the conversation and extracts the durable, salient information — facts about the user, decisions made, preferences, key events — as concise memories, rather than dumping the raw transcript. This keeps memory small, high-signal, and useful to recall later.
Storing raw transcripts recreates the unbounded-growth problem in the store and makes recall noisy. You distill: keep salient facts/events, not every word.
The last message is arbitrary — the important thing to remember might be earlier. You need a step that identifies what's actually salient across the turn/session.
Run a memory writer that extracts the salient information — durable facts about the user, decisions, preferences, key events — usually with an LLM deciding what matters, and stores concise memories rather than the raw transcript. Decide when to write (after each turn, or a summary at session end) and what type (a fact vs an event). Distilling at write time keeps memory small, high-signal, and easy to recall — the opposite of dumping everything.
Why this piece earns its place
The writer is a non-deterministic component writing into durable state, and that combination is unusual enough to plan for explicitly. Change its prompt and you change what the system remembers from now on, but nothing already stored moves. Six months in, the store has strata: memories written under three generations of extractor, with different granularity, different typing conventions, different ideas of what counts as important — and recall ranks across all of them as peers, so a prompt you fixed in March is still being quoted back at the user in September. Stamp each memory with the extractor version that wrote it. That single field is what turns "we improved the prompt" into a job you can actually run: select everything the previous version produced and re-extract it. Without it there is no cohort to select, only an undifferentiated pile, and every bad write is permanent by default. The second-order effect lands on evaluation. You cannot A/B a new extractor on live traffic the way you would a ranker, because the losing variant leaves residue in real users' stores. Shadow-write it into a separate namespace and compare recall quality there before it touches anything.
- extractsalient facts/events
- concisenot the transcript
- whenper turn / session
What the new pieces do
- Memory Writerservice
- After a turn/session, extracts the salient facts worth keeping (not the raw transcript) and writes them to memory.
Step 4 · Build the prompt
Assembling working memory
Every model call needs its context assembled from several sources under a hard token budget. What goes in, and how do you fit it?
A context builder assembles each prompt from: the system prompt, the recent turns (short-term), and the recalled long-term memories relevant to this turn — all packed within the token budget, prioritizing the most important/relevant and truncating or summarizing the rest. This is the moment memory becomes useful: the right past facts are placed into the window right when they're needed, so the stateless model behaves as if it remembers.
Why this piece earns its place
Two things go wrong here that look like tuning and are really design. First, the budget usually gets written as a remainder — system prompt, then recent turns, then recalled memories in whatever space is left. That makes personalization a function of how verbose the last exchange happened to be: a user who pastes a stack trace silently evicts everything the agent knows about them, at exactly the point the conversation got hard. Give each section a reserve it cannot lose and let the recent turns compete for the remainder, not the other way round — and remember the completion comes out of the same window, so a packer that fills to the model limit leaves nowhere for the answer to go. Second, where you place a memory decides how it is treated. Text in the system prompt reads as instruction and is obeyed; the same sentence dropped into a retrieved-context block reads as reference and is weighed against everything else. Nobody decides this deliberately — it gets decided by whoever wrote the string concatenation — and it is the difference between a preference the agent honors and one it merely knows.
- system + recentshort-term
- + recalledlong-term
- token budgetpack by relevance
What the new pieces do
- Context Builderservice
- Assembles each prompt: system + recent turns + recalled long-term memories, all under the token budget.
Step 5 · Where memories live
Vector + structured stores
Memories aren't all the same shape. "The user seemed frustrated last week" (fuzzy, recall by similarity) is different from "the user's name is Sam" (a fact you look up exactly). Where do they go?
Use the right store per memory type. A vector store holds embedded memories for semantic recall ("find memories related to what they're asking now") — great for episodic/fuzzy memory. A structured store / profile holds authoritative facts (name, preferences, settings) queried directly, not by similarity, so they're always correct and current. (Some systems add a knowledge graph for relationships.) Match storage to how each memory is written and recalled.
Why this piece earns its place
Two stores means every write is a small distributed transaction that you do not actually have. The extractor decides a turn updates the profile and deserves an episode; the profile write lands, the vector upsert fails or arrives a second later, and the two halves of memory now disagree with no error anywhere. Two-phase commit is the wrong size of hammer here. Decide the order instead, and pick it so a half-finished write fails in the survivable direction: profile row first, episode second, so an interrupted write costs you the story around a fact while the fact itself stays exactly right. The other order keeps the mood and loses the name, which is the version a user notices immediately. Then make the divergence detectable, because nothing on the request path will ever raise it — the consolidator already passes over the store in the background, so give it a second job: check the two sides against each other, episodes asserting a fact the profile never received, profile fields with nothing behind them. The structured side buys you one more thing — a vocabulary. A model left to fill a free-text field will write vegetarian, veggie and no meat as three values for one key; a constrained enum per field turns contradiction detection into an equality check instead of a judgment call.
- vector storeepisodic / semantic recall
- profileexact facts
- graphrelationships (optional)
What the new pieces do
- Profile / Factsdata
- Structured, authoritative facts about the user (name, preferences, key settings) — queried directly, not by similarity.
- Memory Storedata
- The long-term memory: embedded memories in a vector store, recalled by semantic similarity to the current turn.
Step 6 · Pull the right memories
The recall path
At each turn, of potentially thousands of stored memories, only a few are relevant. Injecting all of them is impossible (token budget) and injecting the wrong ones misleads the model. How do you recall well?
Thousands of memories exist; only a few are relevant now. How do you recall the right ones?
All memories won't fit the token budget and would bury the relevant ones in noise, degrading answers. You retrieve only the few relevant to the current turn.
Recency alone misses relevant older memories and wastes budget on irrelevant recent ones. You want relevance to the current turn (often combined with recency and importance).
Treat the memory store as a corpus: embed the current query/turn, retrieve the top relevant memories by similarity (often blended with recency and an importance score), pull exact facts from the profile, and inject just those into the context. It's retrieval-augmented generation where the corpus is the user's own history.
Recall is RAG over memory. Embed the current turn, retrieve the most relevant memories from the vector store (often blending similarity with recency and an importance score), fetch exact facts from the profile, and inject just those into the context. Of thousands of memories, only the handful that matter now reach the model — enough for continuity, few enough to fit the budget and avoid misleading noise.
Why this piece earns its place
Recall fails in two directions and only one of them is visible. A memory that should have surfaced and did not costs you a generic answer — the user shrugs, repeats themselves, and nothing in your telemetry records that anything happened. A memory that surfaces when it should not is the one that gets screenshotted: the agent volunteering something irrelevant, dated, or from a context the user did not expect it to reach into. Because the two are not symmetric, the ranker needs a floor, not just an ordering. A top-k query always returns k rows; nearest is not the same as relevant, and in a thin store the nearest memory to anything is simply whatever exists. Injecting nothing has to be a legal outcome. Worth saying too that the three signals are not equally trustworthy: recency is computed exactly, relevance is an approximation, and importance was guessed at write time by a model that had not seen the turn it would eventually be recalled for. Weight it accordingly, keep the weights in config where you can move them without a deploy, and keep how many you inject in the request, where a cheap turn and an expensive one can differ.
- embed turnquery memory
- rankrelevance+recency+importance
- injectthe few that matter
What the new pieces do
- Recallservice
- Retrieves the memories relevant to the current turn (RAG over memory) to inject into the context.
Step 7 · Keep it coherent
Consolidation & forgetting
Left alone, memory grows forever, accumulates duplicates, and holds contradictions (the user moved cities; both the old and new fact are stored). More memory then makes recall worse. How do you keep it healthy?
Run consolidation (a background process, like sleep for memory): summarize and merge related memories, deduplicate, resolve contradictions (newer/authoritative facts supersede older ones), and decay/forget stale or low-importance memories so the store stays bounded. This keeps recall precise and memory current — critically, an update to a fact (name change, new preference) must overwrite/supersede, not just add another contradictory memory.
Why this piece earns its place
Consolidation is the only process here that destroys information, and it does it unattended, on a model's judgment, with nobody in the loop to notice. The compounding failure is summary-of-summaries: one pass compresses a dozen episodes into a paragraph, a later pass reads that paragraph as its input and compresses again, and by the third generation something specific the user said has become a vague characterization of them that no longer resembles it. Nothing errors, and the original is gone. Carry a generation count on every memory, refuse to compress past a small number of passes, and pin high-importance items so they are never an input to a summary at all. The other thing to decide is when it runs. A nightly sweep over every user reads the whole store per user per night, so the bill scales with how much you have stored rather than how much traffic you served — a dormant account with two years of history costs the same as your heaviest user. Trigger on writes since the last run, and consolidate within a cluster of related memories rather than across the whole store.
- merge + dedupcompress
- supersederesolve contradictions
- decaybound growth
What the new pieces do
- Consolidatorservice
- Background process that summarizes, merges duplicates, resolves contradictions and decays old memories to keep the store coherent.
Step 8 · The sharp edges
Staleness, privacy & poisoning
Memory has dangerous failure modes: recalling stale/contradictory facts, poor recall precision (right memory not surfaced, or wrong one injected), sensitive PII stored indefinitely, and memory poisoning (a user planting false "facts" the agent later trusts).
Fight staleness with consolidation and by preferring the authoritative profile for facts (and timestamping memories). Improve recall precision with better ranking, importance scoring, and typed memories. Handle privacy: treat memory as sensitive user data — consent, encryption, retention limits, and a "forget me" that truly deletes. Defend against poisoning by not blindly trusting extracted "facts" (especially security-relevant ones), validating/attributing memories, and scoping memory per user. And watch cost — recall + write + consolidation are extra model/DB calls per interaction.
Why this piece earns its place
One control is missing from that list and it is the one that makes the rest operable: let the user see and edit what the agent remembers. Memory bugs are per-user state, so they do not reproduce. You get a report that the assistant keeps insisting on something wrong, and there is no stack trace, no failing request, nothing in the logs that differs from a healthy turn — the pipeline ran perfectly and returned a line that has been wrong since a bad extraction weeks ago. The only person who can point at which stored sentence is the wrong one is the person it is about, which makes a plain list of memories with a delete button the real support tool, and a faster repair than any of the automated defenses. It carries a second-order effect worth accepting deliberately: once memory is visible, every stored line has to survive being read by the person it describes. That does not rule out the inferred memories — a note that someone seemed frustrated last week is exactly what the vector store is for — but it changes how you write them. Store the observation, not the verdict on the person. "Asked three times for shorter answers" is something a user can read and correct; "is impatient" is something they can only resent. Same observation, two forms, and only one of them survives being shown.
You did it
You just designed an agent memory system.
Everything you assembled, in order
- LLMs are stateless — without memory they forget across the window and across sessions.
- Stuffing the whole history fails: it overflows the window, costs more, and never persists across sessions.
- Split memory: short-term (context window) vs long-term (persistent store), moving salient info between them.
- Write path: extract and store salient facts/events (an LLM decides), not the raw transcript.
- Assemble each prompt under a token budget: system + recent turns + recalled memories.
- Store by type (vector store for episodic/semantic recall, a structured profile for exact facts); recall is RAG over memory.
- Consolidate and forget — merge, dedup, resolve contradictions, decay — so memory stays coherent, current and bounded.
