{"notes":[{"id":"transformers/flashattention","category":"transformers","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"FlashAttention","subtitle":"重算比较快","author":"Z. Yuan","paperDate":"2022-05-27","publishedOn":"2023-10-14","paperTitle":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","paperUrl":"https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html","bibliography":{"title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","authors":["Tri Dao","Daniel Y. Fu","Stefano Ermon","Atri Rudra","Christopher Ré"],"url":"https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html","pdfUrl":"https://papers.nips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf","codeUrl":"https://github.com/Dao-AILab/flash-attention","preprint":{"id":"2205.14135","submittedOn":"2022-05-27","url":"https://arxiv.org/abs/2205.14135"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html"},"citation":"@misc{transformersflashattention2022,\n  title = {{FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness}},\n  author = {Tri Dao and Daniel Y. Fu and Stefano Ermon and Atri Rudra and Christopher Ré},\n  year = {2022},\n  eprint = {2205.14135},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.14135}\n}\n","citationPath":"/content/papers/citations/transformers/flashattention.c58786df0331.bib"},"thumbnail":{"src":"/content/papers/assets/0df05e80bbc932893fde60a74e418e29ea95d39830ad1bf354eda2c19415b346.png","width":473,"height":482},"locale":"zh-hans","path":"/content/papers/zh-hans/transformers/flashattention.91df72b672e2.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"先认识这三张表","text":"先认识这三张表","depth":3},{"id":"四个分数分两次算","text":"四个分数，分两次算","depth":3},{"id":"训练时真的再算一次","text":"训练时，真的再算一次","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"算得更多等得更少","text":"算得更多，等得更少","depth":3},{"id":"换张卡答案还一样吗","text":"换张卡，答案还一样吗？","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/transformer","relation":"background","topic":"注意力的内存与计算成本"},{"id":"transformers/flashattention2","relation":"next","topic":"注意力的内存与计算成本"}]},{"id":"transformers/transformer","category":"transformers","legacyPath":"transformers/1706-transformer","legacySlug":"1706-transformer","legacyPaths":[],"title":"Transformer","subtitle":"新世界的起点","author":"Z. Yuan","paperDate":"2017-06-12","publishedOn":"2024-05-05","paperTitle":"Attention Is All You Need","paperUrl":"https://arxiv.org/abs/1706.03762","bibliography":{"title":"Attention Is All You Need","authors":["Ashish Vaswani","Noam Shazeer","Niki Parmar","Jakob Uszkoreit","Llion Jones","Aidan N. Gomez","Lukasz Kaiser","Illia Polosukhin"],"url":"https://arxiv.org/abs/1706.03762","pdfUrl":"https://arxiv.org/pdf/1706.03762","codeUrl":null,"preprint":{"id":"1706.03762","submittedOn":"2017-06-12","url":"https://arxiv.org/abs/1706.03762"},"publication":{"venue":"NeurIPS (NIPS)","year":2017,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html"},"citation":"@misc{transformerstransformer2017,\n  title = {{Attention Is All You Need}},\n  author = {Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin},\n  year = {2017},\n  eprint = {1706.03762},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1706.03762}\n}\n","citationPath":"/content/papers/citations/transformers/transformer.86c0068d96fd.bib"},"thumbnail":{"src":"/content/papers/assets/29a55cb05497bdd9bbb3f3a96b7ed6656fda3552fa6a0fa5eb29e8e899c1acc1.jpg","width":756,"height":1080},"locale":"zh-hans","path":"/content/papers/zh-hans/transformers/transformer.b84de2c39ebe.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型设计","text":"模型设计","depth":3},{"id":"输入层","text":"输入层","depth":3},{"id":"位置编码","text":"位置编码","depth":3},{"id":"自注意力机制","text":"自注意力机制","depth":3},{"id":"多头注意力机制","text":"多头注意力机制","depth":3},{"id":"跨注意力机制","text":"跨注意力机制","depth":3},{"id":"feed-forward-网络","text":"Feed-Forward 网络","depth":3},{"id":"layer-normalization","text":"Layer Normalization","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"为什么要用自注意力机制","text":"为什么要用自注意力机制？","depth":3},{"id":"实验结果机器翻译","text":"实验结果：机器翻译","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/flashattention","relation":"next","topic":"注意力的内存与计算成本"},{"id":"transformers/bert","relation":"next","topic":"双向语言表示的预训练"},{"id":"transformers/gpt_1","relation":"next","topic":"从语言预训练到上下文学习"}]},{"id":"transformers/flashattention2","category":"transformers","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"FlashAttention-2","subtitle":"看 GPU 在发呆","author":"Z. Yuan","paperDate":"2023-07-17","publishedOn":"2024-02-23","paperTitle":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","paperUrl":"https://arxiv.org/abs/2307.08691","bibliography":{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","authors":["Tri Dao"],"url":"https://arxiv.org/abs/2307.08691","pdfUrl":"https://arxiv.org/pdf/2307.08691v1","codeUrl":"https://github.com/Dao-AILab/flash-attention","preprint":{"id":"2307.08691","submittedOn":"2023-07-17","url":"https://arxiv.org/abs/2307.08691"},"publication":{"venue":"ICLR","year":2024,"kind":"conference","url":"https://proceedings.iclr.cc/paper_files/paper/2024/hash/98ed250b203d1ac6b24bbcf263e3d4a7-Abstract-Conference.html"},"citation":"@misc{transformersflashattention22023,\n  title = {{FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning}},\n  author = {Tri Dao},\n  year = {2023},\n  eprint = {2307.08691},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2307.08691}\n}\n","citationPath":"/content/papers/citations/transformers/flashattention2.c97f05c9efd2.bib"},"thumbnail":{"src":"/content/papers/assets/09345c90336d8f77f03ee7897086f96836da6049c67ea85707c77ad183ba6d50.png","width":1320,"height":742},"locale":"zh-hans","path":"/content/papers/zh-hans/transformers/flashattention2.37df8e9b1020.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"沿着行切工作","text":"沿着行切工作","depth":3},{"id":"除法留到最后","text":"除法留到最后","depth":3},{"id":"warp-也要重新排","text":"warp 也要重新排","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"先看注意力本身","text":"先看注意力本身","depth":3},{"id":"再看整场训练","text":"再看整场训练","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/flashattention","relation":"background","topic":"注意力的内存与计算成本"}]}],"text":"写程序时，我们直觉上会把算过的结果先存起来，下次直接拿来用，总比重新算一次划算。\n\n这个直觉到了 GPU 上，怎么好像不太对啊？\n\nGPU 算得很快，但把结果存进内存、之后再读回来，也要花时间。不仅如此，那些搬来搬去的时间，甚至比重新算一遍还久！\n\n这篇论文的作者注意到了这个问题，所以试着在 Attention 上换一种做法，把计算拆成小块，减少数据来回搬运；反向传播需要中间结果时，就在 GPU 的高速内存里重新算。\n\n对，你没看错！\n\n算完直接丢弃，之后有需要的时候，重新算。\n\n## 定义问题\n\n先看下图。四个位置彼此打分，得到 $4\\times4=16$ 格；八个位置变成 $8\\times8=64$ 格。长度加倍，表格面积变四倍。图中只画了一个样本、一个注意力头；「头」先当成一组独立计分的向量就好。\n\n![四个位置有十六格分数，八个位置有六十四格。](/content/papers/assets/113f35026eaa6c3e1caf000469d6dbfead6ee8f7b3751155f9aa735141f7f3b1.png)\n\n每一行代表「这个位置」对整段序列的评分。分数还要经过 softmax，把每个分数 $x$ 变成 $e^x$（$e$ 约为 2.718），再除以这一行的 $e^x$ 总和，权重才会加起来等于 1。后面计算输出时，分数高的位置就占较大的比重。\n\n一般实现会把整张分数表写进 GPU 的大内存 HBM，读回来做 softmax，再把权重表写回去。序列长 2048 时，**光是一个注意力头的一张表**就有 419 万格；用半精度存约 8 MiB（约 8.4 MB）。模型若有多个头、多个样本，还要乘上去。\n\nGPU 芯片内另有 SRAM，容量小得多，但数据送到运算单元的速度快得多。作者想把眼前需要的一小块放进 SRAM，算完就走，不让完整的分数表和权重表在 HBM 来回。\n\n只是 softmax 要求**同一行的所有位置共用一个分母**。你若只看前半行，算出一组加起来为 1 的权重；后半行出现更大的分数时，前半行的权重该怎么改？\n\n## 解决问题\n\n### 先认识这三张表\n\n把一段文字送进注意力层后，每个位置会得到三个向量，$Q$ 是拿来查询的 query，$K$ 是接受比对的 key，$V$ 是最后要依权重汇总的内容 value。假设序列有 $N$ 个位置，每个头的向量长度是 $d$，标准密集注意力写成：\n\n$$\nS=QK^\\mathsf{T}\\in\\mathbb{R}^{N\\times N},\\qquad\nP=\\operatorname{softmax}(S/\\sqrt d),\\qquad O=PV.\n$$\n\n沿着其中一行走，一个 query 跟所有 key 打分，除以 $\\sqrt d$ 避免向量变长时分数容易偏大；softmax 把分数变成权重，再拿这些权重对 value 做加权平均。最后留下的 $O$ 是输出。麻烦的是中途的 $S$ 和 $P$，每张都有 $N^2$ 格。\n\nFlashAttention 把 $Q,K,V$ 切块。原论文 Figure 1 画的就是这件事：\n\n![原论文 Figure 1 中央架构图：Q、K、V 分块搬入 SRAM，再逐块更新输出。](/content/papers/assets/0df05e80bbc932893fde60a74e418e29ea95d39830ad1bf354eda2c19415b346.png \"原论文 Figure 1 中央局部，PDF 第 2 页。\")\n\n左边是 $Q$、上方是 $K$、右边是 $V$。橙色小格表示这次搬进 SRAM 的数据；虚线框是原本可能整张写进 HBM 的注意力表。红箭头沿 $K,V$ 的数据块前进，蓝箭头沿 $Q$ 的数据块前进。每碰到一组，就算一小格分数并更新输出。所有 query 还是会遇到所有 key，**省的是保存与搬运整张中间表**，没有少算那些配对。\n\n> [!TIP]\n>\n> 图上红、蓝箭头的内外层顺序，对应原论文的 [Algorithm 1](https://papers.nips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf#page=5)。先不用记循环；接下来只追一行分数，就能看懂小格之间如何交接。\n\n### 四个分数，分两次算\n\n假设一行已经除过 $\\sqrt d$ 的分数是 $[3,1,4,2]$，对应的 value 在某一个维度是 $[1,0,3,2]$。四个位置一次看完，输出是：\n\n$$\nO=\\frac{e^3\\cdot1+e^1\\cdot0+e^4\\cdot3+e^2\\cdot2}{e^3+e^1+e^4+e^2}\\approx2.3429.\n$$\n\n现在把前两个、后两个分开算。为免 $e^x$ 太大，每组先减去自己的最大分数 $m$。同一组分数都减去同一个数，softmax 的分子和分母会一起缩放，最后答案不变。我们只需留下最大值 $m$、指数总和 $\\ell$，以及还没除以 $\\ell$ 的加权和 $\\nu$。\n\n![两块分数先对齐最大值，再合并指数和与加权和。](/content/papers/assets/764976d63532c1a29ce321d1a05eab920e7cc758666d846b2a1999e1338c8950.png)\n\n左边 $[3,1]$ 减去 3，指数变成 $[1,e^{-2}]$；所以 $\\ell=1+e^{-2}\\approx1.1353$，搭配 value $[1,0]$ 得到 $\\nu=1$。右边 $[4,2]$ 减去 4，也得到 $\\ell\\approx1.1353$，但 value 是 $[3,2]$，所以 $\\nu=3+2e^{-2}\\approx3.2707$。\n\n两边各自都算对了，却不能把 $\\nu$ 直接相加。左边用 3 当基准，右边用 4；直接相加等于把两边最高分当成一样高。于是改用 4 当共同基准，把左边的 $\\ell$ 和 $\\nu$ 都乘上 $e^{3-4}$：\n\n$$\n\\begin{aligned}\n\\ell&=e^{-1}(1.1353)+1.1353\\approx1.5530,\\\\\n\\nu&=e^{-1}(1.0000)+3.2707\\approx3.6386,\\\\\nO&=\\nu/\\ell\\approx2.3429.\n\\end{aligned}\n$$\n\n答案回到一次看完四格的 $2.3429$。第三块来了，就把刚才合并的 $(m,\\ell,\\nu)$ 当作「前面所有块」，照样更新。这就是 online softmax，不存整行分数，也能让不同小格共用最后那个分母。实际的 value 有 $d$ 个维度，所以 $\\nu$ 是向量；每行要带着走的数据仍远小于整张表。\n\n### 训练时，真的再算一次\n\n前向计算从输入走到输出，现在不用保存整张权重表 $P$。但训练还有反向步骤，模型要算出各参数该往哪里调，会再次用到注意力权重。FlashAttention 走到哪一格，就从 $Q,K$ 和前向留下的归一化数据重算那一格。\n\n![一般做法保存完整权重表；FlashAttention 在反向需要时重算小格。](/content/papers/assets/cd328d6bc4b3f016f48c0f66f5ad9d444faad814ffebae76c560a3d2161f8a7d.png)\n\n回到开场那个直觉。重算会增加乘加运算，却省去把整张 $P$ 写入 HBM、再从 HBM 读回来的时间。论文的 [Theorem 2](https://papers.nips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf#page=6) 也把这笔交换写成数据访问量。标准做法是 $\\Theta(Nd+N^2)$，分块后是 $\\Theta(N^2d^2/M)$；$M$ 是 SRAM 能容纳的数值个数，条件是 $d\\le M\\le Nd$。$\\Theta$ 看的是长度增加时的成长趋势，不计固定倍数。\n\n公式告诉我们有机会少搬数据，但**整个计算到底有没有更快？**\n\n## 讨论\n\n### 算得更多，等得更少\n\n来看原论文 [Figure 2 左图](https://papers.nips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf#page=6)。作者在 A100 上测注意力的前向加反向，序列长 1024、每头向量长 64、16 个头、一次处理 64 个样本，有 padding mask、没有 dropout。\n\n![原论文 Figure 2：左图比较算术量、HBM 读写和时间；中图比较分块大小；右图比较稀疏注意力。](/content/papers/assets/b0df1011836ff3018f6bdaa410d1cbe9d8846b17ca64bd93379bc9636e9f797c.png \"原论文 Figure 2，NeurIPS 2022，PDF 第 6 页。\")\n\n| Figure 2 | 一般注意力 | FlashAttention |\n| --- | ---: | ---: |\n| 算术量 | 66.6 GFLOPs | 75.2 GFLOPs |\n| HBM 读写 | 35.3 GB | 4.4 GB |\n| 时间 | 35.1 ms | 11.7 ms |\n\nGFLOPs 是十亿次浮点运算，GB 是搬运的数据量，ms 是千分之一秒。依序读这三行，算术量增加约 13%；HBM 读写量剩约八分之一；时间则从 35.1 ms 降到 11.7 ms。重算花掉的时间，确实比省下的搬运少。\n\n中图则显示，块从 64 加到 256，搬运与时间大致一起下降；加到 512，时间没有照比例再降。SRAM 容量和 GPU 上怎么排工作，都会影响最合适的块大小。\n\n序列拉长后呢？原论文 [Figure 3](https://papers.nips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf#page=9) 左边量时间，右边量注意力使用的内存：\n\n![原论文 Figure 3：不同序列长度下，注意力的运行时间与内存用量。](/content/papers/assets/c549b184d9a5ea0f1af3fd76b42bf842328c36aad46298b3c7f1f8d00bd328b5.png \"原论文 Figure 3，NeurIPS 2022，PDF 第 9 页。\")\n\n右图的 FlashAttention 曲线大致随长度直线增加；一般做法的中间表则跟 $N^2$ 一起长。在论文所测设置下，注意力内存最多差到 20 倍。不过左图的时间仍快速上升，因为 query 和 key 的配对一个都没少。右图延伸到 64K 的棕线是近似方法 Linformer，别把它误看成一般密集注意力。\n\n注意力快了，整个模型会快多少？原论文 [Table 1](https://papers.nips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf#page=7) 的 BERT-large 提供一个答案：\n\n![原论文 Table 1：BERT-large 达到训练目标的时间，原版为 20.0±1.5 分钟，FlashAttention 为 17.4±1.4 分钟。](/content/papers/assets/33f47dc890c47359f5a338886cc192236778adb1400b9494dea9a2383e73314a.png \"原论文 Table 1 摘录，NeurIPS 2022，PDF 第 7 页。\")\n\n达到同一训练目标，时间从 $20.0\\pm1.5$ 分钟降到 $17.4\\pm1.4$ 分钟。模型还有别的层，因此整场训练的降幅小于刚才注意力内核的降幅；这个对照也让 Figure 2 的数字有了落地的位置。\n\n### 换张卡，答案还一样吗？\n\n我们也用 RTX 4090 看了一次。让同一张卡做同样的注意力计算，一边用 PyTorch eager 分三步算，一边用 [v1.0.9 的 Triton 分块前向内核](https://github.com/Dao-AILab/flash-attention/blob/v1.0.9/flash_attn/flash_attn_triton.py)；只改序列长度。设置是 fp16、一次两个样本、8 个头、每头向量长 64，没有遮罩或 dropout；预热后取 30 次计时的中位数。额外内存从 $Q,K,V$ 已备好的时候开始算。\n\n| 序列长度 | PyTorch eager | 分块内核 | 速度比 | 额外峰值：eager / 分块 |\n| ---: | ---: | ---: | ---: | ---: |\n| 512 | 0.0501 ms | 0.0781 ms | 0.64× | 18.0 / 1.06 MiB |\n| 1024 | 0.1341 ms | 0.1044 ms | 1.28× | 68.0 / 2.13 MiB |\n| 2048 | 0.9533 ms | 0.2007 ms | 4.75× | 264.0 / 4.25 MiB |\n\n512 个位置时，分块反而慢；到了 2048，少搬大表才压过分块本身的成本。这里只量前向，反向和整个模型的训练速度要回到论文的实验看。\n\n## 结论\n\n回头看第一代的设计，最值得学习的地方有两个。\n\n首先，**算术量不是速度的全部**。在论文中的 A100 实验里，FlashAttention 多做了约 13% 的运算，Attention 的前向与反向总时间却从 35.1 ms 降到 11.7 ms，只剩三分之一。关键在于，HBM 读写量从 35.3 GB 降到了 4.4 GB。多算一些，换来少搬很多，非常划算。\n\n其次，**中间结果可以「需要时再算」**。online softmax 让不同小块的结果能正确接起来；反向传播则利用保留下来的输出与 softmax 统计量，在 SRAM 里重新计算需要的注意力权重，省掉保存整张权重表的开销。完整的 Attention 照样算，但计算顺序与数据保存方式换了，速度和内存用量就跟着改善。\n\n不过，第一代也还没把 GPU 的算力充分用起来。softmax 更新时的缩放与归一化，仍有可以省下的非矩阵运算；工作主要沿着 batch 和 head 分配，当两者数量较少时，GPU 就可能有一部分算力闲着。即使在同一个 thread block 里，不同 warp 也得通过 shared memory 交换、合并中间结果，付出额外的读写与同步成本。\n\n数据已经搬得少了，能不能再把工作分得更好，让 GPU 少等一点、多算一点？\n\n我们先喝杯咖啡，回来接着看。\n","imageSizes":{"/content/papers/assets/113f35026eaa6c3e1caf000469d6dbfead6ee8f7b3751155f9aa735141f7f3b1.png":{"width":1000,"height":575},"/content/papers/assets/0df05e80bbc932893fde60a74e418e29ea95d39830ad1bf354eda2c19415b346.png":{"width":473,"height":482},"/content/papers/assets/764976d63532c1a29ce321d1a05eab920e7cc758666d846b2a1999e1338c8950.png":{"width":1000,"height":820},"/content/papers/assets/cd328d6bc4b3f016f48c0f66f5ad9d444faad814ffebae76c560a3d2161f8a7d.png":{"width":1000,"height":575},"/content/papers/assets/b0df1011836ff3018f6bdaa410d1cbe9d8846b17ca64bd93379bc9636e9f797c.png":{"width":1260,"height":460},"/content/papers/assets/c549b184d9a5ea0f1af3fd76b42bf842328c36aad46298b3c7f1f8d00bd328b5.png":{"width":1270,"height":460},"/content/papers/assets/33f47dc890c47359f5a338886cc192236778adb1400b9494dea9a2383e73314a.png":{"width":680,"height":104}}}
