FlashAttention and why it matters

FlashAttention (Dao et al., 2022; v2 in 2023; v3 in 2024) is an IO-aware exact attention algorithm that reorders the attention computation to operate on tiles of Q,K,V that fit in on-chip SRAM, then accumulates softmax statistics without materializing the full N x N attention matrix in HBM. Two big wins: (1) memory drops from O(N^2) to O(N), enabling 100k+ context windows; (2) speedups of 2-4x because HBM bandwidth is the dominant cost in attention. FlashAttention-2 improves parallelism and work partitioning; FlashAttention-3 (Hopper) uses async warp-specialization and FP8 to reach ~2x v2. Senior nuance: FlashAttention enabled practical long-context training but it is an "exact" implementation: approximate attention (Sparse, Linear) sometimes beats it for million-token contexts.