Transformer-XL,
explained.

A standard transformer attends over a fixed-length window — a segment of, say, 512 tokens. To process a long document you chop it into segments and run each one independently. That creates two problems. First, the model can never learn a dependency longer than one segment, because information never crosses a boundary — the paper calls this context fragmentation. Second, at evaluation time, to predict each new token you slide the window forward by one and recompute the whole segment from scratch, which is enormously wasteful.

Both problems come from the same root: each segment is an island. Transformer-XL's fix is to build bridges between the islands — let a segment see the one before it — without paying to reprocess the past. Two mechanisms make that work, and they turn out to depend on each other.

The first mechanism is segment-level recurrence. When processing a segment, Transformer-XL caches the hidden states it produced for the previous segment, at every layer. The current segment's attention can then read from its own tokens and from those cached states — extending its view one segment into the past. The cache is treated as fixed input (stop-gradient: no backpropagation flows into it), so training stays cheap.

The elegance is how this compounds with depth. Layer 1 of the current segment reaches back into layer 1 of the previous segment. But layer 2 reads layer-1 states that already incorporated the segment before that. So each additional layer extends the reach by roughly another segment — the effective context grows about linearly with the number of layers, not staying pinned at a single segment length. A 12-layer model with 512-token segments can carry information across thousands of tokens.

Recurrence breaks something. The original transformer adds an absolute position embedding — position 0, 1, 2, … within the segment. But with recurrence, the same absolute indices reappear in every segment: the cached previous segment also has a "position 3." Add absolute embeddings and two tokens from different segments look identically placed; the model can't tell the recent "position 3" from the older one. Recurrence and absolute positions are incompatible.

The fix is relative positional encoding: encode the distance between a query and a key, not their absolute indices. Distance is well-defined across segment boundaries — a token 5 steps back is 5 steps back whether it's in this segment or the last:

Because rel(5,3) = rel(105,103) = 2, the encoding is identical no matter where in the stream the pair sits — exactly the property recurrence needs. Transformer-XL folds this relative distance directly into the attention score computation. The runnable version below confirms the shift-invariance and the context growth.

Recurrence pays a second dividend at inference. A vanilla transformer, generating token by token, slides its fixed window forward one step at a time and recomputes the entire context for each new token — order L work per token for a segment length L. Transformer-XL already has the previous segment's states cached, so it processes a new segment once and reuses the cache — no re-computation of the past.

So the same cache that extends the context also removes the redundant recomputation that made long-context evaluation painfully slow. Longer reach and faster inference from one idea — carry the past forward instead of rebuilding it.

Transformer-XL delivered concrete gains that mattered at the time:

It was, for a period, the standard for long-sequence language modeling, and it seeded XLNet (which built its permutation language modeling on the Transformer-XL backbone). Its ideas became reference points for how to think about memory and position in transformers.

Two of Transformer-XL's contributions outlived the specific model. Relative positional encoding became a lasting theme — the recognition that transformers should reason about distance rather than absolute index runs directly into RoPE and the relative-bias schemes used across modern models. And the idea of a recurrent memory — carrying compressed past state forward instead of re-attending to raw tokens — recurs in memory-augmented and state-space models aiming at very long context.

The broader arc is instructive. Long context has been attacked from several directions: sparse attention (Longformer), efficient exact attention (FlashAttention), and recurrence/memory (Transformer-XL). Each trades differently, and the field still borrows from all three. Transformer-XL's specific answer — recur across segments, encode position relatively — was an early, influential statement that a transformer's context needn't end where its window does.