{"notes":[{"id":"transformers/flashattention2","category":"transformers","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"FlashAttention-2","subtitle":"The GPU is just sitting there","author":"Z. Yuan","paperDate":"2023-07-17","publishedOn":"2024-02-23","paperTitle":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","paperUrl":"https://arxiv.org/abs/2307.08691","bibliography":{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","authors":["Tri Dao"],"url":"https://arxiv.org/abs/2307.08691","pdfUrl":"https://arxiv.org/pdf/2307.08691v1","codeUrl":"https://github.com/Dao-AILab/flash-attention","preprint":{"id":"2307.08691","submittedOn":"2023-07-17","url":"https://arxiv.org/abs/2307.08691"},"publication":{"venue":"ICLR","year":2024,"kind":"conference","url":"https://proceedings.iclr.cc/paper_files/paper/2024/hash/98ed250b203d1ac6b24bbcf263e3d4a7-Abstract-Conference.html"},"citation":"@misc{transformersflashattention22023,\n  title = {{FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning}},\n  author = {Tri Dao},\n  year = {2023},\n  eprint = {2307.08691},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2307.08691}\n}\n","citationPath":"/content/papers/citations/transformers/flashattention2.c97f05c9efd2.bib"},"thumbnail":{"src":"/content/papers/assets/09345c90336d8f77f03ee7897086f96836da6049c67ea85707c77ad183ba6d50.png","width":1320,"height":742},"locale":"en","path":"/content/papers/en/transformers/flashattention2.d4fa5b656ffd.md","headings":[{"id":"the-problem","text":"The problem","depth":2},{"id":"the-method","text":"The method","depth":2},{"id":"assign-work-by-row","text":"Assign work by row","depth":3},{"id":"divide-at-the-end","text":"Divide at the end","depth":3},{"id":"reassign-the-warps","text":"Reassign the warps","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"measure-attention-first","text":"Measure attention first","depth":3},{"id":"then-measure-the-whole-training-run","text":"Then measure the whole training run","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/flashattention","relation":"background","topic":"Attention memory and computation"}]},{"id":"transformers/flashattention","category":"transformers","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"FlashAttention","subtitle":"Recompute to go faster","author":"Z. Yuan","paperDate":"2022-05-27","publishedOn":"2023-10-14","paperTitle":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","paperUrl":"https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html","bibliography":{"title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","authors":["Tri Dao","Daniel Y. Fu","Stefano Ermon","Atri Rudra","Christopher Ré"],"url":"https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html","pdfUrl":"https://papers.nips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf","codeUrl":"https://github.com/Dao-AILab/flash-attention","preprint":{"id":"2205.14135","submittedOn":"2022-05-27","url":"https://arxiv.org/abs/2205.14135"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html"},"citation":"@misc{transformersflashattention2022,\n  title = {{FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness}},\n  author = {Tri Dao and Daniel Y. Fu and Stefano Ermon and Atri Rudra and Christopher Ré},\n  year = {2022},\n  eprint = {2205.14135},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.14135}\n}\n","citationPath":"/content/papers/citations/transformers/flashattention.c58786df0331.bib"},"thumbnail":{"src":"/content/papers/assets/0df05e80bbc932893fde60a74e418e29ea95d39830ad1bf354eda2c19415b346.png","width":473,"height":482},"locale":"en","path":"/content/papers/en/transformers/flashattention.157a71142542.md","headings":[{"id":"the-problem","text":"The problem","depth":2},{"id":"the-method","text":"The method","depth":2},{"id":"meet-the-three-matrices","text":"Meet the three matrices","depth":3},{"id":"four-scores-two-passes","text":"Four scores, two passes","depth":3},{"id":"during-training-calculate-it-again","text":"During training, calculate it again","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"more-arithmetic-less-waiting","text":"More arithmetic, less waiting","depth":3},{"id":"does-another-gpu-give-the-same-answer","text":"Does another GPU give the same answer?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/transformer","relation":"background","topic":"Attention memory and computation"},{"id":"transformers/flashattention2","relation":"next","topic":"Attention memory and computation"}]}],"text":"FlashAttention used tiling, online softmax, and recomputation to keep the attention matrix from making repeated trips through HBM. It substantially reduced memory use and made the computation faster, too.\n\n- [**[22.05] FlashAttention: Recompute to go faster**](/blog/papers/transformers/flashattention)\n\nBut when we open the GPU's work queue, something looks wrong.\n\nWhy is it...\n\nso idle?\n\n## The problem\n\nWe all know that work is divided at two levels inside a GPU.\n\nA GPU has many Streaming Multiprocessors (SMs) that can work at the same time. A program groups threads into a thread block and assigns the block to an SM. Within a block, threads are divided into warps, usually 32 threads per warp.\n\nThink of an SM as a factory, a thread block as a work order, and a warp as one of the crews doing the work.\n\nThe first FlashAttention mainly assigns work according to batch size × number of attention heads. Take the A100 from the paper: it has 108 SMs. If batch size is 1 and there are 8 attention heads, we get only 8 thread blocks.\n\nA hundred and eight factories. Eight work orders.\n\nWhat do the other hundred do? Take a nap, apparently.\n\nA longer sequence only makes those eight orders thicker; it does not bring more factories online. The paper reports that the first implementation reaches only 25–40% of theoretical peak throughput. There is plenty of room left on the GPU.\n\nAnd that is not all.\n\nTensor Cores are excellent at matrix multiplication, but softmax requires many operations that are not matrix multiplications. These apparently simple operations run much less efficiently and slow the calculation down. Meanwhile, inside a first-generation block, different warps each handle part of $K,V$. They then exchange partial results through shared memory, synchronize, and merge them.\n\nTo sum up the problem so far:\n\n> Too little work is distributed among SMs, and the work inside each block is divided poorly among warps.\n\nCould we simply cut the work into smaller pieces and run more SMs at once?\n\nAttention does not split that easily. Every score in a softmax row needs the same normalization information. During backpropagation, separate blocks may also need to add to the same gradient. Cut too coarsely and the GPU sits idle; cut too finely and everyone waits for everyone else.\n\nHow do we give everyone work without spending the day in meetings?\n\nThat is the problem FlashAttention-2 sets out to solve.\n\n## The method\n\nThe authors split the forward work in two places, with different thread blocks owning different groups of query rows and different warps within a block owning their own query rows. After that split, each row must still be able to finish its softmax. Let us follow one row through the arrangement.\n\n### Assign work by row\n\nOf the three letters on the score table, $Q$ says what we are looking for, $K$ provides the clues to compare against, and $V$ holds the content we eventually retrieve. Imagine attention scores as a table. Its rows come from $Q$, its columns from $K$, and each cell scores one pairing.\n\nSoftmax runs across a row. Scores in the same row share a denominator, while a different row has its own denominator. The authors therefore **assign a group of complete rows to one thread block**, which reads the $K,V$ tiles it needs in sequence and produces those rows' outputs.\n\nLook at the paper's [Figure 2](https://arxiv.org/pdf/2307.08691v1#page=8). In the left panel, each color marks a group of query rows assigned to one worker (a thread block). The white cells are skipped because of the causal mask. When generating text, earlier words must not look into the future, leaving the upper-right corner of the table empty.\n\n![Paper Figure 2: forward workers are assigned query rows; backward workers are assigned K and V columns.](/content/papers/assets/09345c90336d8f77f03ee7897086f96836da6049c67ea85707c77ad183ba6d50.png \"Excerpt from paper Figure 2, arXiv:2307.08691v1, PDF page 8.\")\n\nThe first worker computes the first group of rows, the second worker the next group. Both may read the same $K,V$ tile, but they need not exchange intermediate softmax results. In our earlier example with one batch and eight heads, each head performs its own attention calculation. If we split each head's query rows into 16 groups, eight work orders become $8\\times16=128$. Sixteen is just an illustration; the actual count depends on tile size. Either way, a longer sequence can now finally create more pieces of parallel work.\n\nWhy does the right panel switch to columns? The backward pass computes $dQ,dK,dV$, the gradients sent back to $Q,K,V$. A $K,V$ tile receives contributions from many query rows. Giving one worker a column lets it accumulate that column's $dK,dV$ itself.\n\n$dQ$, in contrast, receives contributions from different columns. Those workers use atomic adds when updating $dQ$, so simultaneous writes do not overwrite one another. Forward workers own output rows; backward workers own $K,V$ gradients.\n\n\n> [!TIP]\n> \n> The paper credits Phil Tillet's Triton implementation with first changing the forward loop to visit query rows before key columns and parallelizing it along the sequence. FlashAttention-2 combines that arrangement with the calculation and warp changes below.\n\n\n### Divide at the end\n\nA worker assigned some query rows still has to read $K,V$ tile by tile. But the softmax denominator is not known until we have seen every score in the row. Could we divide the partial output after every tile? Yes; the first generation does. But if the next tile contains a higher score, we must rescale and divide again.\n\nLet us work through a few numbers.\n\nSplit one row of scores into $[3,1]$ and $[4,2]$, with corresponding $V$ values $[1,0]$ and $[3,2]$. To keep exponentials under control, use the largest score seen so far, $m$, as a reference. Also keep the sum of exponentials $\\ell$ and the weighted sum $U$, before division.\n\n![Original teaching diagram based on paper Algorithm 1: the first tile leaves m, ℓ, and U; the second changes the reference maximum; only the last step divides U by ℓ.](/content/papers/assets/263e97203044605ebc4462d97e2f80f38f576a5e33bfc3f6304cdea519ee7c3e.png \"DOCSAID diagram based on FlashAttention-2 Algorithm 1: two score tiles and a single final division, O = U / ℓ.\")\n\nStart with the first tile in the diagram. Its maximum score is $3$, so $\\ell_1=e^{3-3}+e^{1-3}=1+e^{-2}\\approx1.1353$. Its weighted sum is $U_1=1\\times1+e^{-2}\\times0=1$. The first generation would calculate a partial output, $U_1/\\ell_1\\approx0.8808$, at this point. The second generation keeps $\\ell_1$ and $U_1$ without dividing.\n\nThe second tile brings a higher score, $4$. Old values used $3$ as their reference. To put everything on the new scale, multiply the old $\\ell_1,U_1$ by $e^{3-4}=e^{-1}$. Add the second tile to obtain $\\ell_2=e^{-1}\\ell_1+1+e^{-2}\\approx1.5530$ and $U_2=e^{-1}U_1+3+2e^{-2}\\approx3.6386$. Only after both tiles are done do we divide: $O=U_2/\\ell_2\\approx2.3429$. Softmax over the whole row gives the same answer.\n\nIn general, let $S_j$ be a newly read tile of scores and $V_j$ its values. A prime marks the state after that tile:\n\n$$\n\\begin{aligned}\nm'&=\\max(m,\\max S_j),\\\\\n\\ell'&=e^{m-m'}\\ell+\\sum e^{S_j-m'},\\\\\nU'&=e^{m-m'}U+e^{S_j-m'}V_j.\n\\end{aligned}\n$$\n\nThe factor $e^{m-m'}$ is the $e^{-1}$ in our diagram, which moves the old denominator and weighted sum onto the new reference scale. The remaining terms add the new tile's exponentials to $\\ell'$ and its value-weighted contribution to $U'$.\n\nIn real attention, $V_j$ contains vectors, so $U$ is a vector too. The diagram uses single numbers so we can work it out by hand. Each query row maintains its own $m,\\ell,U$ and computes $O=U/\\ell$ only when it is finished.\n\nThe backward pass must recompute attention weights. The authors save just one normalization number per row, $L=m+\\log\\ell$. Since $e^L=e^m\\ell$, a score $S_{ij}$ has weight $P_{ij}=e^{S_{ij}-L_i}$. Scores are still recomputed, but there is no need to retain the full weight matrix or save $m$ and $\\ell$ separately.\n\nWhy bother saving a few divisions? The paper lists theoretical A100 throughput of 312 TFLOPs/s for FP16/BF16 matrix multiplication and 19.5 TFLOPs/s for ordinary FP32 operations. Reducing the latter leaves more time for the matrix multiplications Tensor Cores do best.\n\n### Reassign the warps\n\nMore thread blocks give more SMs work. Now open one of those blocks and look at its warps. In the paper's [Figure 3](https://arxiv.org/pdf/2307.08691v1#page=9), orange dashed outlines mark data split among warps; blue dashed outlines mark data available to all of them.\n\n![Paper Figure 3: the first generation splits K and V among warps; the second splits Q so warps own different output rows.](/content/papers/assets/3f883e1b61fb2b5b3921a5868057cf51f8024140fa46d1cade2011bd06cb174a.png \"Excerpt from paper Figure 3, arXiv:2307.08691v1, PDF page 9.\")\n\nOn the left, four warps share the same $Q$, but each gets part of $K,V$. When they finish, each holds only **part of the same output rows**. To deliver complete rows, they write partial results to shared memory, wait, read them back, and combine them. The paper calls this split-K.\n\nThe right panel reverses the division, letting every warp use $K,V$ while $Q$ is split four ways. The first warp produces its own output rows, the second produces the next group. Each output row has one owner, so the forward pass no longer needs a shared-memory meeting to piece it together. Backward gradients still have dependencies and require some synchronization.\n\nThe two levels now line up. A head's query rows are distributed among more blocks, and each block's rows are divided among its warps. Within each row, the softmax division waits until the end.\n\n## Discussion\n\n### Measure attention first\n\nThe paper's [Figure 4(a)](https://arxiv.org/pdf/2307.08691v1#page=10) compares attention **forward plus backward** on an A100 80GB SXM4, without a causal mask and with a head dimension of 64. The horizontal axis is sequence length. The vertical axis is throughput in TFLOPs/s, calculated using the paper's FLOP formula; at the same setting, a higher bar means a faster run. Orange marks the first generation and purple the second.\n\n![Paper Figure 4(a): forward-plus-backward throughput on A100, without a causal mask and with head dimension 64.](/content/papers/assets/4da8f5a0a33326ba067ed325ee8fb91124c226bf64fa0583340eb9b77b3de460.png \"Excerpt from paper Figure 4(a), arXiv:2307.08691v1, PDF page 10.\")\n\nAt 8k, the first generation reaches 110 TFLOPs/s and the second 175, a ratio of $175/110\\approx1.59$. At 16k the numbers are 110 and 176. PyTorch is marked OOM at the far right because it ran out of memory in that setting.\n\nFor this chart, the authors hold the total number of tokens per test at 16k, so longer sequences mean a smaller batch, a point that ties back to the opening. At 8k the batch is 2; at 16k it is 1.\n\nThis chart uses 32 heads. At 16k, the first generation has only $1\\times32=32$ pieces of work from batch and head, facing 108 SMs on the A100. Splitting the query rows into additional blocks addresses that shortage of parallel work.\n\n### Then measure the whole training run\n\nA faster attention layer is one thing. What about the whole model? The paper's [Table 1](https://arxiv.org/pdf/2307.08691v1#page=12) trains GPT-style models on eight A100s, comparing no FlashAttention, the first generation, and the second. The 1.3B and 2.7B labels are parameter counts; 2k and 8k are context lengths in tokens. Within one row of the table, a higher TFLOPs/s per GPU means higher training throughput.\n\n![Paper Table 1: per-GPU training throughput for GPT-style models on eight A100s with three attention implementations.](/content/papers/assets/04a80f79e2aa4937e06424841176b888d22415d4a406f4fa628ab29189f23ba8.png \"Excerpt from paper Table 1, arXiv:2307.08691v1, PDF page 12.\")\n\nRead across the two 1.3B rows. At 2k context, the first generation scores 189 and the second 196 TFLOPs/s, an increase of about 4%. At 8k, it goes from 170 to 220, about 29%. The 2.7B model at 8k similarly rises from 175 to 225, about 29%. Without FlashAttention, that same row reaches only 80, making the second generation about 2.8 times faster.\n\nIn this table, the shorter context gains relatively little while longer-context training gains more. That fits the reason for adding parallel work along the sequence.\n\nFinally, the authors turn the peak 225 TFLOPs/s into 72% model FLOPs utilization. Utilization of the 108 SMs is a separate hardware metric. Table 1 divides a model-FLOP estimate by elapsed time. Following the convention used in earlier work, its formula does not subtract the roughly half of the attention score table omitted by causal masking. The numbers can compare training throughput under the same row's conditions.\n\n## Conclusion\n\nThe first FlashAttention addressed **too much data movement**. The second asks **how to distribute the work**.\n\nIt launches more thread blocks along the query sequence so more SMs can work at once, rearranges warps to reduce their exchanges, and changes the order of online-softmax operations to save some work outside matrix multiplication.\n\nThe design later became an acceleration option in [**PyTorch SDPA**](https://docs.pytorch.org/blog/pytorch2-2/) and [**Hugging Face**](https://huggingface.co/docs/transformers/main/llm_optims). Still, dense attention grows quadratically with sequence length, and the best arrangement for an A100 need not be the best on an H100 or B200.\n\nAfter all, **if everyone looks busy, does that mean the whole production line is efficient?**\n\nNext time, the H100 takes the floor.\n","imageSizes":{"/content/papers/assets/09345c90336d8f77f03ee7897086f96836da6049c67ea85707c77ad183ba6d50.png":{"width":1320,"height":742},"/content/papers/assets/263e97203044605ebc4462d97e2f80f38f576a5e33bfc3f6304cdea519ee7c3e.png":{"width":1000,"height":850},"/content/papers/assets/3f883e1b61fb2b5b3921a5868057cf51f8024140fa46d1cade2011bd06cb174a.png":{"width":1440,"height":525},"/content/papers/assets/4da8f5a0a33326ba067ed325ee8fb91124c226bf64fa0583340eb9b77b3de460.png":{"width":736,"height":517},"/content/papers/assets/04a80f79e2aa4937e06424841176b888d22415d4a406f4fa628ab29189f23ba8.png":{"width":1403,"height":210}}}
