FlashAttention
An exact attention algorithm that avoids ever materializing the full attention matrix in slow GPU memory.
Standard attention implementations write the full N×N attention matrix out to GPU high-bandwidth memory, which is slow and memory-hungry at long sequence lengths. FlashAttention restructures the computation to work in small tiles that stay in the GPU’s much faster on-chip SRAM, computing the exact same mathematical result without ever writing the full matrix out — a pure systems-level speedup, not an approximation.
Worked example: an exact-attention algorithm that avoids materializing the full N×N attention matrix in slow GPU memory — it tiles the computation and fuses the softmax so it stays in fast on-chip SRAM, making long-context attention much faster and lower-memory without approximation. Gotcha: it is a systems win (memory-IO-bound, not compute-bound), not a math change — outputs are identical to standard attention; the speedup comes from respecting the GPU memory hierarchy, the same lesson as PagedAttention.