Titans,
explained.

Rung 4's state is a compressed summary written by fixed weights. This rung asks a different question: what if the long-term memory is itself a small network whose weights are updated during inference, so that reading a long input literally trains the module that will later be asked to recall it? The memory stops being a buffer the model writes into and becomes a model the input fits.

The interesting part is what decides whether something is written, because a memory that stores everything is a transcript again. The rule is surprise: how much the memory module would have to change to account for this input is exactly how much it did not already predict it, and that quantity — the gradient of the memory's own loss — is used as the write signal. Unsurprising input barely moves the weights; a genuinely new fact moves them a lot. Two further controls keep that usable at length: a momentum term, so a surprising passage keeps influencing the update for a while rather than only at the instant it appeared, and explicit forgetting, so finite capacity is reclaimed instead of saturating.

Stated in the units the ladder compares on: the long-term memory is consulted at a cost that does not scale with how much has been stored, so this is a third answer to rung 4's capacity problem — not a bigger fixed state and not exact attention, but a learned map. The architecture is presented with several ways to wire it in relative to the attention that remains — as extra context, as a gate, or as its own layer — and the reported evaluations run to contexts in the millions of tokens.

Attention is a phenomenal short-term memory: every token can look at every other token, losslessly. The price is quadratic cost and a hard window — outside it, the past simply doesn't exist. Recurrent architectures like Mamba take the opposite trade: compress all history into a fixed-size state. Linear cost, unlimited length — but compression is lossy, and details fade.

Model weights are the third store: a vast long-term memory of everything seen in training — and completely frozen at inference. Nothing you show a deployed model changes them. So the field's dilemma: perfect-but-bounded recall, or unbounded-but-lossy compression, with no memory that grows during use. Titans aims squarely at that gap.

Titans adds a neural long-term memory: a small MLP whose whole job is to store associations — and whose weights are updated during inference. As the sequence streams in, each chunk can trigger a gradient step on the memory module. Reading is training, for that one small component; the backbone stays frozen.

Retrieval is just a forward pass: query the memory MLP and it reproduces what it stored. Because the memory is weights — not a token buffer — its capacity doesn't consume the context window, and reading a 2M-token document doesn't mean attending over 2M tokens. The distant past lives in the memory; attention handles the recent window it's actually good at.

A memory that writes everything equally would drown in routine text. Titans borrows a page from human memory: we remember what violates expectations. Its write strength is driven by a surprise metric — the gradient of the memory's prediction error on the incoming input. Content the memory already anticipates yields small gradients: barely stored. Content that breaks its expectations yields large gradients: written hard.

The elegance is that the signal is free — it's the same gradient the update already computes. Surprise decides what to remember; the gradient step decides how. One mechanism, both jobs.

Raw surprise has failure modes: a single shocking token shouldn't erase context (surprise spikes then vanishes), and a finite memory that never forgets eventually overwrites itself into mush. Titans handles both with tools straight out of optimization.

Momentum smooths the write signal: past surprise carries into the present, so an important passage keeps getting written even when individual tokens stop being shocking. A learned forgetting gate (weight decay on the memory) fades stale associations to make room — adaptive, data-dependent forgetting rather than a fixed horizon. The memory module is, in effect, a tiny optimizer with momentum and decay, running inside inference.

Line it up and it's literally SGD-with-momentum, run over the token stream — the surprise is the gradient of the memory's loss, and the write steps against it (gradient descent):

Momentum β carries a surprise forward; the forget gate decays what isn't refreshed. The runnable version below writes a secret code hard, a routine line faintly, and watches memory fade under silence.

The paper tests three placements of the memory relative to attention:

Alongside both sits a small set of persistent memory tokens — learnable, input-independent parameters holding task knowledge. Three stores, three timescales: persistent (trained, fixed), long-term (written at test time), short-term (attention over the window).

Titans reopened a question the field had parked: maybe long context isn't a window problem but a memory problem. Instead of racing to attend over ever-more tokens, let part of the model learn its input. That framing — test-time learning as an architectural primitive — has rippled into follow-up architectures, memory research, and the agent world's obsession with what to remember across long horizons.

It also completes a neat symmetry with the reasoning era: test-time compute spends more thinking per query; Titans spends learning per query. Both push intelligence out of the frozen-training paradigm and into the moment of use — which is likely where the next architectural decade gets decided.

Titans, from Ali Behrouz, Peilin Zhong and Vahab Mirrokni at Google Research, adds a neural long-term memory to an attention-based model. The memory is a small MLP whose weights are updated by gradient descent while the model reads, on an online associative-memory loss.

1. Project each token to a key and a value. k_t = x_tW_K and v_t = x_tW_V. Following recent linear recurrent models, the projections are followed by a short depthwise convolution, and keys and queries are ℓ2-normalized.
2. Measure surprise. The memory M is asked to map the key to the value, with loss ‖M(k_t) − v_t‖². The gradient of that loss with respect to M's weights is the momentary surprise: large when the memory could not predict this association, small when it could.
3. Accumulate surprise with momentum. S_t = η_tS_t−1 − θ_t∇ℓ. Past surprise decays at rate η_t instead of vanishing, so a surprising passage keeps being written for a while after the token that triggered it.
4. Forget, then write. M_t = (1 − α_t)M_t−1 + S_t. The forgetting gate α_t acts as weight decay on the memory and frees capacity. All three controls, η_t, θ_t and α_t, are computed from the input rather than fixed.
5. Read without writing. Retrieval is a plain forward pass with a query, y_t = M*(q_t) with q_t = x_tW_Q. No weights change on a read.
6. Make it trainable in parallel. Updating weights token by token is sequential, so the sequence is split into chunks. Within a chunk every gradient is taken with respect to the memory's weights at the start of the chunk, a mini-batch form of the update that can be written as matrix multiplications, and the momentum term is combined across the chunk with a parallel scan.
7. Add persistent memory. A set of learnable, input-independent tokens is prepended to the sequence to hold task knowledge. With attention as short-term memory and the neural module as long-term memory, that makes three stores.
8. Wire it in one of three ways. In memory as context (MAC), each segment's tokens query the memory, the recalled vectors are concatenated with the persistent tokens and the segment, attention runs over that window, and the attention output is written into the memory and used to gate the final output. In memory as gate (MAG), sliding-window attention and the memory run as parallel branches combined by a gate. In memory as a layer (MAL), the memory is a layer applied before sliding-window attention. The memory alone, without attention, is also tested as a recurrent model.

Where it sits among test-time learners

Weights used as fast-changing memory go back to fast-weight programmers, and linear attention can be read as one. DeltaNet's delta rule is the special case of this update with a linear memory, no momentum and no forgetting. Test-time training (TTT) layers (Sun et al., 2024) also train an inner model by gradient descent on a self-supervised loss during the forward pass. Titans adds the momentum term, data-dependent forgetting and a deeper MLP memory, and combines the result with attention.

Training setup

The language models were trained on FineWeb-Edu at roughly 340M, 400M and 760M parameters, on 15 to 30 billion tokens, against Transformer++ and recurrent baselines including Mamba, Mamba-2, DeltaNet, Gated DeltaNet and TTT.

• M — the memory: the weights of a small MLP, one or more layers deep.
• ℓ — the associative-memory loss: how badly the memory maps this token's key to its value.
• ∇ℓ — momentary surprise, the gradient with respect to the memory's weights.
• St — accumulated surprise, a momentum buffer; ηt sets how fast past surprise decays.
• θt — how strongly this token's surprise is written, effectively a learning rate.
• αt — the forgetting gate, a weight decay between 0 and 1.
• M*(q) — a read: a forward pass with no update.

At the extremes: α_t = 1 wipes the memory, and α_t = 0 never forgets. η_t = 0 removes momentum, leaving one plain gradient step per token, the form TTT layers use. θ_t = 0 turns writing off. With a linear memory, a matrix, and η = α = 0, the gradient is 2(M k_t − v_t)k_t^^⊤ and the update becomes the delta rule of DeltaNet.

The paper compares its three variants, and the memory module alone, against Transformer++ and a field of modern recurrent models at matched sizes, then tests long-context recall separately.

The ablations test each part of the update. Replacing the deep memory with a linear one, removing the convolution, removing momentum, removing the forgetting term and removing the persistent tokens each made results worse. Deeper memory modules improved perplexity but lowered training throughput, a trade-off the paper reports directly.

Two cautions. The language-model comparisons stop below a billion parameters, a scale at which new sequence primitives have often looked better than they later did. And the long-context results lean on synthetic and semi-synthetic benchmarks such as needle-in-a-haystack and BABILong, which reward exactly the kind of fact storage the memory is built for. Beating far larger general-purpose models such as GPT-4 on BABILong shows that the mechanism suits the benchmark; it does not by itself show a general advantage.

This is recent work, and the right register for it is what it would take to believe rather than what it has proved. The mechanism's appeal is real: a surprise-gated write is a principled answer to what a fixed-capacity memory should keep, and it is the same intuition that makes prioritised replay work, applied to storage instead of sampling.

The questions it raises are mostly about serving rather than quality, and they are substantial. A module whose weights change while processing a request is stateful in a way current inference stacks are not built for: two requests cannot share it without one contaminating the other, batching is complicated by every sequence wanting its own copy, and the same input can produce different outputs depending on what the module saw before — which is a problem for reproducibility, for caching, and for anything that has to be audited. None of these is unanswerable; none is answered by the architecture.

What would settle it: independent reproduction at scale, and a comparison against the strong hybrid baselines from rung 4 under matched compute rather than against a plain transformer. Long-context benchmarks are also unusually easy to overfit to — a synthetic retrieval task can be solved by a mechanism that helps nothing else — so the result to watch for is whether the gains show up on tasks nobody designed with this memory in mind.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Attention costs quadratic in sequence length. Six attempts to escape that, ending with two 2026 papers that give the model a memory instead.

1. 1Transformer-XL2019
2. 2Longformer2020
3. 3Ring Attention2023
4. 4Mamba2023
5. 5Titans2025
6. 6Engram2026

All 110 papers, in 19 ladders →