FlashAttention,
explained.

Self-attention has every token compare itself to every other token, producing an N×N scores matrix. For a sequence of length N, that matrix grows with the square of N — at 8,000 tokens it's 64 million numbers. The standard implementation writes that whole matrix out to memory, reads it back to apply softmax, then reads it a third time to multiply by the values.

The paper's diagnosis: the GPU's arithmetic units were mostly idle, waiting on those reads and writes. Attention was memory-bound, not compute-bound. The field had been optimizing the wrong thing.

A GPU has two very different kinds of memory. HBM is the big main memory — tens of gigabytes, but relatively slow. SRAM is tiny on-chip memory — only kilobytes to a few megabytes, but roughly an order of magnitude faster.

Standard attention parks that giant scores matrix in slow HBM and keeps trudging back to it. FlashAttention's entire strategy is to never put it there in the first place.

Instead of computing the full matrix at once, FlashAttention breaks the work into tiles — small blocks of queries and keys that fit inside fast SRAM. It loads a block, computes its piece of attention entirely in SRAM, and moves on, never writing the intermediate scores back to HBM.

The tricky part is softmax, which normally needs to see a whole row at once to normalize it. FlashAttention uses an online softmax: it processes blocks one at a time and keeps a running running-total and running-maximum, rescaling previous partial results as new blocks arrive — so it gets the exact normalized answer without ever holding the full row. All the attention operations are fused into a single pass over fast memory.

This is what set FlashAttention apart. Earlier attempts to speed up attention approximated it — sparse patterns, low-rank tricks — trading accuracy for speed. FlashAttention changes only the memory schedule. The result is bit-for-bit the same attention; you give up nothing.

FlashAttention delivered large end-to-end speedups — on the order of 2–4x faster attention and meaningful whole-model training speedups — while cutting attention memory from quadratic to linear in sequence length. It made previously impractical context lengths routine.

FlashAttention (and its successors, FlashAttention-2 and -3) became a default building block baked into every major training and inference stack. If you've used a model with a long context window, you've almost certainly used it.

Its deeper lesson is one every systems engineer learns eventually: on modern hardware, data movement is the cost, not arithmetic. The fastest code isn't the one that does the least math — it's the one that moves the least memory. FlashAttention is that principle, applied to the single most important operation in modern AI.