Ring attention,
explained.

FlashAttention already made attention memory-linear in sequence length on a single device — a huge win. But "linear on one device" still has a ceiling: eventually the sequence's activations, keys, and values won't fit in one accelerator's memory, no matter how clever the kernel. If you want a context of millions of tokens, one chip is simply not enough memory.

The natural move is to use more devices — but attention is all-to-all: every query must see every key, and naively that means every device needs all the keys, which puts you right back at the memory wall. Ring Attention's insight is that a device doesn't need all the keys at once. It can see them one block at a time, as long as, over the course of the computation, each block passes by. Arrange that traffic as a ring, and you get exact full attention with per-device memory that shrinks as you add devices.

Take a sequence of N tokens and P devices. Split the sequence into P contiguous blocks and give one block to each device — so each holds N/P tokens' worth of queries, keys, and values. This is sequence parallelism: the sequence dimension itself is distributed, unlike the more common splitting of the model's layers or tensors.

Each device is responsible for computing the attention output for its own query block. To do that correctly, its queries must attend to the keys and values of every block, including those living on other devices. The question is how to bring those remote keys and values to each device without anyone having to store them all. The answer is the ring.

Connect the devices in a circle. Each device starts by attending its query block to its own key/value block, then passes its key/value block to the next device and receives one from the previous device. Repeat. After P such rotations, every key/value block has visited every device, so every query block has attended to the entire sequence — and the attention outputs, accumulated blockwise with an online softmax (the FlashAttention trick), are exact.

The key property is the memory: at any instant a device holds only its own block plus the one rotating block — memory of order N/P, not N. So with twice the devices you can handle twice the context at the same per-device memory. Nothing is approximated: it computes the same exact dense attention a single giant device would, just spread out in space and time. The runnable version below makes the scaling and step count concrete.

Passing blocks around a ring means network traffic, and network traffic is usually the enemy of distributed speed. Ring Attention's second trick is to make that traffic free by overlapping it with computation. While a device computes attention between its queries and the current key/value block, it is simultaneously sending that block onward and receiving the next one. The transfer happens in the shadow of the math.

Because attention compute per block grows with the block size while the transfer is a fixed data movement, you can choose a block large enough that computation dominates — and then the ring's communication costs nothing in wall-clock time. That overlap is what makes scaling context across many devices practical rather than bottlenecked on the network.

The consequence is a context length that grows with your hardware budget, not your single chip:

Ring Attention (and its blockwise-transformers foundation) is a big part of why million-token context windows became feasible. It reframed long context as a distributed-systems problem with a clean solution, rather than a hard architectural limit — you buy context length with devices.

Long context has been pursued along different axes, and Ring Attention occupies the systems one. Longformer changes what's attended to (sparsity); Transformer-XL adds recurrence and memory; FlashAttention reorganizes the computation on one device. Ring Attention keeps the attention exact and dense and instead distributes it — and it composes with FlashAttention, using the same online-softmax accumulation blockwise across the ring.

Its lasting idea is that some ceilings are about where computation happens, not what it computes. The attention that wouldn't fit on one device fits fine when spread across many, if you route the data cleverly and hide the movement. That "distribute the sequence, rotate the state, overlap the transfer" pattern is now a standard tool for scaling context, and a reminder that a memory wall is often a systems problem in disguise.