Ring attention,
explained.

Rung 2 buys length by computing something other than attention. This rung buys length while computing exactly the same attention, by changing where the computation lives. The insight it builds on is that attention does not actually need the whole key-value set in memory at once: the softmax can be accumulated block by block, carrying a running normaliser, so a query can be scored against the sequence in pieces and the result is the full attention, exact up to floating-point rounding.

Given that, the change is a distribution strategy. Shard the sequence across a ring of devices, so each device holds one block of queries and one block of keys and values. Every device computes the attention of its queries against the block it currently holds, then passes its key-value block to the next device and receives one from the previous. After a full rotation, every query has been scored against every key, and no device ever held more than its own slice.

Stated in units the ladder can compare: memory per device becomes proportional to sequence length divided by device count, so maximum context scales with the number of devices — add hardware, get length, with no change to the model or the mathematics. And the block hand-off is a communication that can be overlapped with the arithmetic of the current block, so on a fast interconnect the transfer is close to hidden behind compute it would otherwise wait for. This is what puts million-token contexts within reach.

FlashAttention already made attention memory-linear in sequence length on a single device — a huge win. But "linear on one device" still has a ceiling: eventually the sequence's activations, keys, and values won't fit in one accelerator's memory, no matter how clever the kernel. If you want a context of millions of tokens, one chip is simply not enough memory.

The natural move is to use more devices — but attention is all-to-all: every query must see every key, and naively that means every device needs all the keys, which puts you right back at the memory wall. Ring Attention's insight is that a device doesn't need all the keys at once. It can see them one block at a time, as long as, over the course of the computation, each block passes by. Arrange that traffic as a ring, and you get exact full attention with per-device memory that shrinks as you add devices.

Take a sequence of N tokens and P devices. Split the sequence into P contiguous blocks and give one block to each device — so each holds N/P tokens' worth of queries, keys, and values. This is sequence parallelism: the sequence dimension itself is distributed, unlike the more common splitting of the model's layers or tensors.

Each device is responsible for computing the attention output for its own query block. To do that correctly, its queries must attend to the keys and values of every block, including those living on other devices. The question is how to bring those remote keys and values to each device without anyone having to store them all. The answer is the ring.

Connect the devices in a circle. Each device starts by attending its query block to its own key/value block, then passes its key/value block to the next device and receives one from the previous device. Repeat. After P such rotations, every key/value block has visited every device, so every query block has attended to the entire sequence — and the attention outputs, accumulated blockwise with an online softmax (the FlashAttention trick), are exact.

The key property is the memory: at any instant a device holds only its own block plus the one rotating block — memory of order N/P, not N. So with twice the devices you can handle twice the context at the same per-device memory. Nothing is approximated: it computes the same exact dense attention a single giant device would, just spread out in space and time. The runnable version below makes the scaling and step count concrete.

Passing blocks around a ring means network traffic, and network traffic is usually the enemy of distributed speed. Ring Attention's second trick is to make that traffic free by overlapping it with computation. While a device computes attention between its queries and the current key/value block, it is simultaneously sending that block onward and receiving the next one. The transfer happens in the shadow of the math.

Because attention compute per block grows with the block size while the transfer is a fixed data movement, you can choose a block large enough that computation dominates — and then the ring's communication costs nothing in wall-clock time. That overlap is what makes scaling context across many devices practical rather than bottlenecked on the network.

Ring Attention, from Liu, Zaharia and Abbeel at UC Berkeley, runs exact attention, and the rest of each transformer block, over a sequence too long for any one accelerator. It builds on their Blockwise Parallel Transformer, which computes attention and the feed-forward network block by block so that full-length activations are never materialized.

1. Shard the sequence. Split N tokens into P contiguous blocks of c = N/P tokens and give one block to each of P hosts arranged in a ring. Each host computes the queries, keys and values for its own block.
2. Start with the local block. Each host computes attention between its query block and its own key-value block, keeping running statistics per query row in the FlashAttention style: the running maximum score, the running softmax denominator and the running weighted sum of values.
3. Rotate and overlap. While it computes, each host sends its current key-value block to the next host and receives one from the previous host. On the next step it folds the new block into its running statistics. After P steps every query block has met every key-value block.
4. Normalize. Dividing the running sum by the running denominator gives the attention output. It equals full attention up to floating-point rounding: the same terms are added, only in a different order.
5. Keep the feed-forward local. The feed-forward network is position-wise, so each host applies it blockwise to its own tokens with no communication at all.
6. Run the backward pass on the same ring. Gradients use the same rotation, with key-value blocks and their gradient accumulators passed between neighbours, so per-host memory stays proportional to the block size rather than the sequence length.
7. Size the block so communication hides. A block's attention costs about 4dc² floating-point operations, while sending its keys and values moves about 4cd bytes at 16-bit precision. The transfer is hidden when the computation takes longer, which reduces to c ≥ F/B, where F is a host's FLOP rate and B its link bandwidth. It is an arithmetic-intensity condition: work grows with c², traffic only with c.

The causal-mask catch

In a decoder, the causal mask means a query block needs only the key blocks at or before it. With a contiguous split, the host holding the first block has almost nothing to compute, while the host holding the last block attends to every block. Because the ring advances in lockstep, every step waits for the busiest host, and roughly half of all compute slots sit idle. Striped Attention (Brandon et al., 2023) fixes this by dealing tokens to hosts round-robin, so each host holds a mix of early and late positions. The zigzag split used in later context-parallel training, including the scheme described for Llama 3, gives each host one early chunk and its mirror-image late chunk.

How it differs from DeepSpeed-Ulysses

DeepSpeed-Ulysses (Jacobs et al., Microsoft, 2023) also shards the sequence, but before attention it runs an all-to-all exchange that re-shards by attention head, so each device computes full-length attention for a subset of heads, then exchanges back. Its parallelism is capped by the number of heads, and each attention still spans the whole sequence on one device. Ring Attention has no head limit and never gathers the full sequence anywhere, at the cost of P sequential steps that depend on the overlap condition.

• Qi — this host's query block, c rows of dimension d; it never moves.
• Kb, Vb — the key-value block passing through this host at step s.
• m — the running maximum score per row, starting at −∞; subtracting it keeps every exponential bounded.
• ℓ, O — the running softmax denominator and the running unnormalized output, rescaled by e^^{m − m′} whenever the maximum rises.
• F, B — a host's FLOP rate and its link bandwidth in bytes per second.
• c — the block size in tokens, the one knob the overlap condition constrains.

At the extremes: with P = 1 the loop runs once and this is ordinary blockwise attention on one device. With c below F/B, every step waits on the network and the ring runs at link speed. Far above it, communication is fully hidden but each host needs more memory. Taking an A100 at 312 teraFLOP/s with about 300 GB/s of NVLink bandwidth, c ≥ about 1,000 tokens; over a 12.5 GB/s network the same chip needs about 25,000.

The authors trained LLaMA-architecture models of several billion to tens of billions of parameters on A100 GPU clusters, connected by NVLink within a node and InfiniBand across nodes, and on TPU v3, v4 and v5e slices. The main comparison is the longest context each method can train at a given model size and hardware: a vanilla transformer, memory-efficient attention, and memory-efficient attention plus a blockwise feed-forward network, which is their earlier Blockwise Parallel Transformer.

The headline claim is a scaling law in hardware: the maximum context is proportional to the number of devices, because each device's memory depends on the block size, not on the total length. That is how the authors reach contexts in the millions of tokens.

Three negatives are worth stating. Total arithmetic is unchanged and still quadratic in sequence length; the ring removes the memory ceiling, not the FLOPs. The overlap argument is a hardware precondition, and on slow interconnects the block size it demands can exceed what fits on a device. And the original paper does not address the causal-mask load imbalance; Striped Attention and zigzag splits came afterwards to recover the roughly half of compute a contiguous causal split leaves idle.

The clearest thing to be honest about is what this does not buy. Total arithmetic is unchanged and still quadratic — the ring distributes and overlaps the work, it does not reduce it. What it removes is the memory ceiling, which was the binding constraint, and that is a genuine and large win, but a million-token forward pass remains a million-token forward pass in FLOPs.

The overlap argument also carries a hardware precondition: it holds when inter-device bandwidth is high enough that a block transfer fits inside a block of compute. On a weaker interconnect the ring stalls and the elegance evaporates, which makes this a technique whose value depends on the cluster it runs on rather than a property of the algorithm.

How it held up: as infrastructure rather than as a paper people cite in product copy. Sequence and context parallelism of this shape is standard in frontier training stacks, and it is part of why long-context models became trainable at all rather than merely fine-tunable. Read alongside rung 2, the pair make the ladder's central engineering point: the two techniques that actually delivered long context both kept attention exact and changed how it meets the memory hierarchy, while the one that changed the mathematics to save operations did not survive.

The consequence is a context length that grows with your hardware budget, not your single chip:

Ring Attention (and its blockwise-transformers foundation) is a big part of why million-token context windows became feasible. It reframed long context as a distributed-systems problem with a clean solution, rather than a hard architectural limit — you buy context length with devices.

Long context has been pursued along different axes, and Ring Attention occupies the systems one. Longformer changes what's attended to (sparsity); Transformer-XL adds recurrence and memory; FlashAttention reorganizes the computation on one device. Ring Attention keeps the attention exact and dense and instead distributes it — and it composes with FlashAttention, using the same online-softmax accumulation blockwise across the ring.

Its lasting idea is that some ceilings are about where computation happens, not what it computes. The attention that wouldn't fit on one device fits fine when spread across many, if you route the data cleverly and hide the movement. That "distribute the sequence, rotate the state, overlap the transfer" pattern is now a standard tool for scaling context, and a reminder that a memory wall is often a systems problem in disguise.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Attention costs quadratic in sequence length. Six attempts to escape that, ending with two 2026 papers that give the model a memory instead.

1. 1Transformer-XL2019
2. 2Longformer2020
3. 3Ring Attention2023
4. 4Mamba2023
5. 5Titans2025
6. 6Engram2026

All 110 papers, in 19 ladders →