Titans,
explained.

Attention is a phenomenal short-term memory: every token can look at every other token, losslessly. The price is quadratic cost and a hard window — outside it, the past simply doesn't exist. Recurrent architectures like Mamba take the opposite trade: compress all history into a fixed-size state. Linear cost, unlimited length — but compression is lossy, and details fade.

Model weights are the third store: a vast long-term memory of everything seen in training — and completely frozen at inference. Nothing you show a deployed model changes them. So the field's dilemma: perfect-but-bounded recall, or unbounded-but-lossy compression, with no memory that grows during use. Titans aims squarely at that gap.

Titans adds a neural long-term memory: a small MLP whose whole job is to store associations — and whose weights are updated during inference. As the sequence streams in, each chunk can trigger a gradient step on the memory module. Reading is training, for that one small component; the backbone stays frozen.

Retrieval is just a forward pass: query the memory MLP and it reproduces what it stored. Because the memory is weights — not a token buffer — its capacity doesn't consume the context window, and reading a 2M-token document doesn't mean attending over 2M tokens. The distant past lives in the memory; attention handles the recent window it's actually good at.

A memory that writes everything equally would drown in routine text. Titans borrows a page from human memory: we remember what violates expectations. Its write strength is driven by a surprise metric — the gradient of the memory's prediction error on the incoming input. Content the memory already anticipates yields small gradients: barely stored. Content that breaks its expectations yields large gradients: written hard.

The elegance is that the signal is free — it's the same gradient the update already computes. Surprise decides what to remember; the gradient step decides how. One mechanism, both jobs.

Raw surprise has failure modes: a single shocking token shouldn't erase context (surprise spikes then vanishes), and a finite memory that never forgets eventually overwrites itself into mush. Titans handles both with tools straight out of optimization.

Momentum smooths the write signal: past surprise carries into the present, so an important passage keeps getting written even when individual tokens stop being shocking. A learned forgetting gate (weight decay on the memory) fades stale associations to make room — adaptive, data-dependent forgetting rather than a fixed horizon. The memory module is, in effect, a tiny optimizer with momentum and decay, running inside inference.

The paper tests three placements of the memory relative to attention:

Alongside both sits a small set of persistent memory tokens — learnable, input-independent parameters holding task knowledge. Three stores, three timescales: persistent (trained, fixed), long-term (written at test time), short-term (attention over the window).

Across language modeling and reasoning benchmarks, Titans outperformed both Transformer baselines and modern recurrent models (Mamba-class) at comparable sizes. The headline demo is needle-in-a-haystack at extreme length: retrieving planted facts with high accuracy in contexts beyond 2 million tokens — lengths where windowed attention can't even see the needle and compressed states have long since blurred it.

Just as telling: on the BABILong benchmark (reasoning across facts scattered through book-length text), a modest Titans model beat far larger models, including ones wielding retrieval. Learned test-time memory wasn't just cheaper than a giant window — it was better at using the far past.

Titans reopened a question the field had parked: maybe long context isn't a window problem but a memory problem. Instead of racing to attend over ever-more tokens, let part of the model learn its input. That framing — test-time learning as an architectural primitive — has rippled into follow-up architectures, memory research, and the agent world's obsession with what to remember across long horizons.

It also completes a neat symmetry with the reasoning era: test-time compute spends more thinking per query; Titans spends learning per query. Both push intelligence out of the frozen-training paradigm and into the moment of use — which is likely where the next architectural decade gets decided.