Engram,
explained.

The whole ladder below this rung treats long context as a cost problem. This rung names a quality problem that persists even when you can afford the compute, and it follows from what softmax does. Attention distributes a fixed total weight across every position, so as the context grows, the one passage that matters is competing against more and more irrelevant ones. Its share of the attention mass is diluted by their number — which is why recall of a specific buried fact degrades with length even in a model whose window nominally covers it.

The change is to add a path that is looked up rather than computed. Alongside attention and the feed-forward layers, the model gets a very large table of learned embeddings, indexed by hashes of the last few tokens. Each lookup costs the same whatever the length of the context, and a context-aware gate decides how much of what comes back to use. The memory is conditional in the way mixture-of-experts computation is conditional: a big store of parameters, of which each position touches only a sliver.

Stated in the units this ladder is argued on, the claim is about the recall metric rather than about throughput: on needle-in-a-haystack retrieval the reported move is from the mid-eighties to the high nineties. That is the same weakness rung 4 admits — a fixed state cannot hold arbitrary detail — attacked from the opposite side. Mamba compresses and hopes the summary kept the right thing; this gives the model a lookup for the local, static patterns it would otherwise have to rebuild in its early layers, and spends the freed capacity on everything else.

Softmax attention answers a query by spreading a fixed budget of probability across every token it can see. That's fine for a short context. But the needle-in-a-haystack test — hide one crucial fact in a long document and ask for it — exposes the flaw: as the haystack grows, the needle's slice of the attention budget shrinks, and it can get washed out by thousands of irrelevant tokens.

Two costs in one: you pay O(n) to scan everything, and the answer quality drops with length. Engram attacks the second one head-on.

Engram adds a conditional memory: a very large table of learned embedding vectors that the model reads by hashing N-grams of the most recent tokens. At each position the last few token IDs — first normalized so that equivalent tokens share an ID — are hashed into slots of the table, using several independent hash functions per N-gram order so that two unrelated N-grams rarely collide in every one of them. The retrieved vectors are combined and passed on.

The table is trained with the rest of the model during pre-training; nothing is written into it while the model runs. Because the address comes from token IDs alone, a lookup costs the same at position 10 as at position 100,000, and it is known before the layer that needs it executes — which is what lets the tables sit in host memory and be prefetched instead of occupying accelerator memory.

A hashed N-gram is a blunt key. The same three tokens mean different things in different sentences, and a collision can put an unrelated pattern in the same slot. So the retrieved memory is not added to the model blindly: the current hidden state acts as a query against it, and a context-aware gate scales the contribution down when the memory disagrees with what the model has already worked out from the context.

"Conditional memory" is the paper's name for this, by analogy with mixture-of-experts. MoE adds capacity that is computed sparsely — each token runs a few experts out of many. Engram adds capacity that is looked up sparsely — each position reads a few slots out of a huge table. The paper treats the split of a fixed parameter budget between the two as a design choice to measure, not to assume.

The long-context gain is indirect, and it is worth being precise about it. The model does not store the needle in the table at inference time — the table is fixed after training. The argument is that once local pattern-reconstruction is offloaded to the lookup, attention has more of its capacity left for finding and using distant context. DeepSeek reports multi-query needle-in-a-haystack accuracy climbing from about 84.2 to 97 with Engram, alongside gains on knowledge, reasoning, code and math benchmarks at matched compute.

What a single retrieval benchmark cannot show is how much of that comes from the memory itself versus from the extra parameters it adds; that is the comparison to look for in independent replications.

Engram is part of a clear shift: stop forcing every piece of knowledge a model holds to be recomputed through stacks of matrix multiplies, and give the model a cheap way to look up what is static. It sits in a line of memory-layer ideas — product-key memories, hashed n-gram embeddings — but pairs the lookup with a context gate and studies it as a second sparsity axis next to mixture-of-experts.

The broader lesson is that "long context" is two problems wearing one name: the cost of a big window and the reliability of what the model does inside it. Sparse and linear attention attack the first; freeing capacity, as Engram does, is one way to attack the second. Compare it with in-context memory designs such as Titans, which write to a memory while the model runs, and with MemGPT, which manages memory outside the model entirely.

Engram is the conditional-memory module described in DeepSeek's 2026 technical report. It gives a transformer a very large table of learned embeddings that is read by lookup instead of by computation. At each position the last few tokens are hashed into the table, and a gate driven by the model's own hidden state decides how much of what comes back to use. The table is learned in pretraining; nothing is written into it at inference time.

1. Form the suffix N-grams. At position t, take the N-grams that end at t: the last two tokens, the last three, and so on up to a small maximum order. Per the report, token IDs are first normalized so that surface variants of the same word share entries.
2. Hash each N-gram with several heads. A table with a row for every possible N-gram would be astronomically large, so each N-gram order has K independent hash functions, each mapping the N-gram to a row of its own embedding table. Two N-grams can collide in one head; colliding in all K heads is very unlikely.
3. Look up and concatenate. The rows from every head and every order are concatenated into one memory vector e_t. The cost is a fixed number of row reads per token whatever the table's size: O(1) in the number of stored parameters, and deterministic.
4. Gate it by context. The same N-gram means different things in different contexts, and a hashed row may be a collision. So e_t is projected to a key and a value, and a scalar gate, a sigmoid of the scaled dot product between the normalized hidden state and the normalized key, scales the value before it joins the residual stream. The hidden state is the query: when the retrieved memory does not fit the context, the gate closes.
5. Place it at a few layers. Per the report, the module sits at a small number of layers rather than all of them, inside a model that also uses mixture-of-experts layers.
6. Train it with the model. The tables are ordinary parameters, learned end to end by the language-modeling loss during pretraining. Each token updates only the few rows it read, so a huge table adds parameters without adding much computation.
7. Serve it from host memory. The row indices depend only on the input token IDs, so every row a forward pass will need is known before it reaches the Engram layers. Those rows can be fetched from CPU memory while earlier layers run, so the table does not have to fit in GPU memory. The report states that the throughput cost of doing this is small.

Why it is called conditional memory

Mixture-of-experts is conditional computation: a huge pool of parameters, of which each token activates only a few experts. Engram is presented as a second axis of the same kind of sparsity, conditional memory: a huge table of which each token reads only a few rows, selected by its local N-gram context. The report treats the two as complementary and studies how to split a fixed parameter budget between them.

What it is not

Engram is not an associative store that writes facts during inference, and it is not retrieval over the document in the context window. Its keys are local token patterns, so it holds what pretraining learned about those patterns, such as names and fixed phrases, and hands it to every position in constant time, which spares the network from rebuilding that knowledge layer by layer.

• xt−n+1, …, xt — the suffix N-gram of order n ending at position t, as normalized token IDs.
• φn,k — hash head k for order n: a fixed, deterministic function of the token IDs alone.
• En,k[z] — row z of that head's embedding table, a trained parameter.
• et — the memory vector: every head's row, for every order, concatenated.
• ĥt, k̂t — the normalized hidden state at this layer and the normalized memory key; the hidden state is the query.
• αt — a scalar gate between 0 and 1: how much of the memory this context accepts.

At the extremes: α near 0 means the context rejects what was retrieved, the right outcome for a hash collision or a phrase used in an unusual sense; α near 1 means the memory is taken in full. With K = 1 a single collision corrupts the vector, while with several heads an N-gram is only fully confused with another if they collide in every head. A small table makes collisions common and leaves more work to the gate; a very large one is what pushes the tables into host memory.

All results below are as reported by DeepSeek in the Engram technical report, and none had been independently reproduced when it appeared. The comparisons hold total parameters and per-token compute fixed, so Engram is tested as a way to spend sparse parameters, not as free extra capacity.

The allocation result is the report's central argument. If memory and experts were interchangeable, the best split would sit at one end; a minimum in the middle says each does something the other does poorly, with experts for computation that depends on context and memory for static local patterns.

The long-context result needs its mechanism stated correctly. Engram's keys are local N-grams at the current position, so it cannot fetch a fact planted thousands of tokens earlier by its content. The report's explanation is indirect: by taking over the reconstruction of local, static patterns, the memory frees attention and early layers for global context, and the report presents layer-by-layer analyses suggesting the network behaves as if it were deeper.

The open questions are the ones that separate a mechanism from a benchmark result: whether the gains hold under independent replication and at other scales, how much of the needle-in-a-haystack gain survives on tasks whose answers are not copyable spans, and how collisions and gating behave in languages and domains that pretraining covered thinly.

Very recent, so the honest summary is that the diagnosis is stronger than the verdict. The diagnosis — that softmax dilution is a distinct failure from compute cost, and that a benchmark measuring average next-token quality will not reveal it — is clearly right and clarifies why the five rungs below disagree: three of them were solving cost and two were solving capacity, and dilution is neither.

What a single retrieval benchmark cannot tell you is whether the memory helps when the useful fact is not a verbatim string. Needle-in-a-haystack tasks reward exact lookup, which is the regime an addressed memory is best at by construction; the harder and more valuable question is multi-hop reasoning over several buried facts, and aggregation over many passages, where what to look up is not given. A mechanism can move a haystack score a long way and leave that untouched.

What would settle it: independent evaluation on tasks whose answers are not copyable spans, and the serving numbers — a conditional memory's value depends entirely on how often the condition fires and what a miss costs, and that is an engineering measurement rather than a claim. Worth reading this rung primarily for the framing: context length and context usability are different quantities, and most reported context windows are measurements of the first.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Attention costs quadratic in sequence length. Six attempts to escape that, ending with two 2026 papers that give the model a memory instead.

1. 1Transformer-XL2019
2. 2Longformer2020
3. 3Ring Attention2023
4. 4Mamba2023
5. 5Titans2025
6. 6Engram2026

All 110 papers, in 19 ladders →