FlashAttention
An exact attention algorithm that avoids ever materializing the full attention matrix in slow GPU memory.
Standard attention implementations write the full N×N attention matrix out to GPU high-bandwidth memory, which is slow and memory-hungry at long sequence lengths. FlashAttention restructures the computation to work in small tiles that stay in the GPU’s much faster on-chip SRAM, computing the exact same mathematical result without ever writing the full matrix out — a pure systems-level speedup, not an approximation.