{"notes":[{"id":"transformers/pre-ln","category":"transformers","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Pre-LN Transformer","subtitle":"不用预热了","author":"Z. Yuan","paperDate":"2020-02-12","publishedOn":"2026-03-18","world":"external/transformer-layernorm","paperTitle":"On Layer Normalization in the Transformer Architecture","paperUrl":"https://arxiv.org/abs/2002.04745","bibliography":{"title":"On Layer Normalization in the Transformer Architecture","authors":["Ruibin Xiong","Yunchang Yang","Di He","Kai Zheng","Shuxin Zheng","Chen Xing","Huishuai Zhang","Yanyan Lan","Liwei Wang","Tie-Yan Liu"],"url":"https://arxiv.org/abs/2002.04745","pdfUrl":"https://arxiv.org/pdf/2002.04745v2","codeUrl":null,"preprint":{"id":"2002.04745","submittedOn":"2020-02-12","url":"https://arxiv.org/abs/2002.04745"},"publication":null,"citation":"@misc{transformerspreln2020,\n  title = {{On Layer Normalization in the Transformer Architecture}},\n  author = {Ruibin Xiong and Yunchang Yang and Di He and Kai Zheng and Shuxin Zheng and Chen Xing and Huishuai Zhang and Yanyan Lan and Liwei Wang and Tie-Yan Liu},\n  year = {2020},\n  eprint = {2002.04745},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2002.04745}\n}\n","citationPath":"/content/papers/citations/transformers/pre-ln.419d25cfd84b.bib"},"thumbnail":{"src":"/content/papers/assets/f44d59bcadead95b9b93a42fc681bc473b038c13ed0c5c9e1effc7ee90409fc3.png","width":436,"height":531},"locale":"zh-hans","path":"/content/papers/zh-hans/transformers/pre-ln.b9202d90578f.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"两种摆法","text":"两种摆法","depth":3},{"id":"预热到底有多重要","text":"预热到底有多重要？","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"看初始化的那一刻","text":"看初始化的那一刻","depth":3},{"id":"第一步主干上的信号有多大","text":"第一步：主干上的信号有多大？","depth":3},{"id":"第二步layernorm-会把梯度除以输入的大小","text":"第二步：LayerNorm 会把梯度除以输入的大小","depth":3},{"id":"第三步连起来","text":"第三步：连起来","depth":3},{"id":"实际量一次","text":"实际量一次","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"翻译不预热还更快","text":"翻译：不预热，还更快","depth":3},{"id":"bert省下四成","text":"BERT：省下四成","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/transformer","relation":"referenced","topic":""},{"id":"transformers/sparse-transformer","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""}]},{"id":"transformers/transformer","category":"transformers","legacyPath":"transformers/1706-transformer","legacySlug":"1706-transformer","legacyPaths":[],"title":"Transformer","subtitle":"新世界的起点","author":"Z. Yuan","paperDate":"2017-06-12","publishedOn":"2024-05-05","paperTitle":"Attention Is All You Need","paperUrl":"https://arxiv.org/abs/1706.03762","bibliography":{"title":"Attention Is All You Need","authors":["Ashish Vaswani","Noam Shazeer","Niki Parmar","Jakob Uszkoreit","Llion Jones","Aidan N. Gomez","Lukasz Kaiser","Illia Polosukhin"],"url":"https://arxiv.org/abs/1706.03762","pdfUrl":"https://arxiv.org/pdf/1706.03762","codeUrl":null,"preprint":{"id":"1706.03762","submittedOn":"2017-06-12","url":"https://arxiv.org/abs/1706.03762"},"publication":{"venue":"NeurIPS (NIPS)","year":2017,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html"},"citation":"@misc{transformerstransformer2017,\n  title = {{Attention Is All You Need}},\n  author = {Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin},\n  year = {2017},\n  eprint = {1706.03762},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1706.03762}\n}\n","citationPath":"/content/papers/citations/transformers/transformer.86c0068d96fd.bib"},"thumbnail":{"src":"/content/papers/assets/29a55cb05497bdd9bbb3f3a96b7ed6656fda3552fa6a0fa5eb29e8e899c1acc1.jpg","width":756,"height":1080},"locale":"zh-hans","path":"/content/papers/zh-hans/transformers/transformer.b84de2c39ebe.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型设计","text":"模型设计","depth":3},{"id":"输入层","text":"输入层","depth":3},{"id":"位置编码","text":"位置编码","depth":3},{"id":"自注意力机制","text":"自注意力机制","depth":3},{"id":"多头注意力机制","text":"多头注意力机制","depth":3},{"id":"跨注意力机制","text":"跨注意力机制","depth":3},{"id":"feed-forward-网络","text":"Feed-Forward 网络","depth":3},{"id":"layer-normalization","text":"Layer Normalization","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"为什么要用自注意力机制","text":"为什么要用自注意力机制？","depth":3},{"id":"实验结果机器翻译","text":"实验结果：机器翻译","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/flashattention","relation":"next","topic":"注意力的内存与计算成本"},{"id":"transformers/bert","relation":"next","topic":"双向语言表示的预训练"},{"id":"transformers/gpt_1","relation":"next","topic":"从语言预训练到上下文学习"}]},{"id":"transformers/sparse-transformer","category":"transformers","legacyPath":"transformers/1904-sparse-transformer","legacySlug":"1904-sparse-transformer","legacyPaths":[],"title":"Sparse Transformer","subtitle":"稀疏注意力机制","author":"Z. Yuan","paperDate":"2019-04-23","publishedOn":"2024-06-09","paperTitle":"Generating Long Sequences with Sparse Transformers","paperUrl":"https://arxiv.org/abs/1904.10509","bibliography":{"title":"Generating Long Sequences with Sparse Transformers","authors":["Rewon Child","Scott Gray","Alec Radford","Ilya Sutskever"],"url":"https://arxiv.org/abs/1904.10509","pdfUrl":"https://arxiv.org/pdf/1904.10509","codeUrl":null,"preprint":{"id":"1904.10509","submittedOn":"2019-04-23","url":"https://arxiv.org/abs/1904.10509"},"publication":null,"citation":"@misc{transformerssparsetransformer2019,\n  title = {{Generating Long Sequences with Sparse Transformers}},\n  author = {Rewon Child and Scott Gray and Alec Radford and Ilya Sutskever},\n  year = {2019},\n  eprint = {1904.10509},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1904.10509}\n}\n","citationPath":"/content/papers/citations/transformers/sparse-transformer.961330a17a72.bib"},"thumbnail":{"src":"/content/papers/assets/847fd8e03beeb4395ae629700c5daaf1fa75dd302949008b9c248f7d77a8f9bc.jpg","width":1392,"height":694},"locale":"zh-hans","path":"/content/papers/zh-hans/transformers/sparse-transformer.1d71fe4e4865.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"最前面的几层","text":"最前面的几层","depth":3},{"id":"第-1920-层","text":"第 19~20 层","depth":3},{"id":"不特定几层","text":"不特定几层","depth":3},{"id":"第-64128-层","text":"第 64~128 层","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"解题思路-1-固定步长","text":"解题思路 1: 固定步长","depth":3},{"id":"解题思路-2-固定特定-index-的-token","text":"解题思路 2: 固定特定 Index 的 Token","depth":3},{"id":"修改后注意力机制","text":"修改后注意力机制","depth":3},{"id":"其他优化","text":"其他优化","depth":3},{"id":"训练设置","text":"训练设置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"cifar-10","text":"CIFAR-10","depth":3},{"id":"enwik8","text":"EnWik8","depth":3},{"id":"imagenet-64x64","text":"ImageNet 64x64","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/longformer","relation":"comparison","topic":"长序列注意力的计算取舍"},{"id":"transformers/bigbird","relation":"comparison","topic":"长序列注意力的计算取舍"}]},{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"捷径要干净","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnet-v2.9e686e0339ca.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"两个恒等映射","text":"两个恒等映射","depth":3},{"id":"捷径上多一点东西会怎样","text":"捷径上多一点东西会怎样？","depth":3},{"id":"相加之后的-relu-该放在哪","text":"相加之后的 ReLU 该放在哪？","depth":3},{"id":"换个角度看pre-activation","text":"换个角度看：pre-activation","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"越深差距越大","text":"越深，差距越大","depth":3},{"id":"和其他方法比较","text":"和其他方法比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]}],"text":"训练 Transformer 有一个奇怪的习惯：学习率不能一开始就设到位。\n\n原始的 Transformer 论文用的调度是这样的：前 4000 步，学习率从接近 0 慢慢爬升到最大值，之后再逐步衰减。这段爬升叫做**学习率预热**（warm-up）。卷积网络通常一开始就用较大的学习率，再慢慢调小；只有像超大批量训练这类特殊情况，才需要预热。Transformer 却几乎每次都要。\n\n预热很麻烦。它让训练变慢，又多了两个要调的超参数：最大学习率和预热步数，而且最终的成绩对这两个值很敏感。训练 BERT 这样的大模型时，每试一组设置都很贵。\n\n为什么 Transformer 非预热不可？2020 年 2 月，中国科学院、北京大学、微软亚洲研究院与南开大学的 Ruibin Xiong、Yunchang Yang、Di He 等人发布了这篇论文，后来发表在 ICML 2020。他们的答案是：问题出在 LayerNorm 放的位置。\n\n\n> [!TIP]\n> \n> 原始 Transformer 每个子层的结构，可以先看我们的笔记；那篇笔记在 LayerNorm 一节也提到，LayerNorm 的位置还可以再区分 Norm-First 与 Norm-Last：\n> \n> - **[[17.06] Transformer：新世界的起点](/blog/papers/transformers/transformer)**\n\n\n## 定义问题\n\n### 两种摆法\n\nTransformer 的每一层有两个子层：自注意力和前馈网络（FFN）。每个子层都有残差连接，也都有 LayerNorm。这三样东西的顺序，可以有不同的排法：\n\n[![左右两个 Transformer 层的示意图。(a) 输入 x_l 经过 Multi-Head Attention，与捷径相加后做 Layer Norm；再经过 FFN，与捷径相加后再做 Layer Norm，得到 x_l+1。(b) 输入 x_l 先做 Layer Norm，再进入 Multi-Head Attention，与捷径相加；接着先做 Layer Norm，再进入 FFN，再与捷径相加，得到 x_l+1。两边的捷径都画成灰色的粗箭头](/content/papers/assets/f44d59bcadead95b9b93a42fc681bc473b038c13ed0c5c9e1effc7ee90409fc3.png)](/content/papers/assets/f44d59bcadead95b9b93a42fc681bc473b038c13ed0c5c9e1effc7ee90409fc3.png)\n\n*原论文 Figure 1，取自 [**v2 PDF 第 2 页**](https://arxiv.org/pdf/2002.04745v2#page=2)。(a) Post-LN Transformer 层；(b) Pre-LN Transformer 层。*\n\n左边是原始 Transformer 和 BERT 的做法：子层、残差相加、LayerNorm，依次进行。LayerNorm 放在残差块**之间**，作者把它叫做 **Post-LN**。以 FFN 子层为例：\n\n$$\nx_{l+1} = \\mathrm{LN}\\big(x + \\mathrm{FFN}(x)\\big)\n$$\n\n右边把 LayerNorm 搬进残差分支里，放在子层**之前**，叫做 **Pre-LN**：\n\n$$\nx_{l+1} = x + \\mathrm{FFN}\\big(\\mathrm{LN}(x)\\big)\n$$\n\nPre-LN 的主干上只有相加；因为主干一路都没有归一化，最后输出前还要再补一层 LayerNorm。\n\n在这篇论文之前就有人用过 Pre-LN：Baevski 与 Auli 的语言模型、OpenAI 的 [Sparse Transformer](/blog/papers/transformers/sparse-transformer) 都采用了它，tensor2tensor、OpenNMT 等翻译工具也有实现；Wang 等人还发现，层数变多时 Pre-LN 比 Post-LN 好。不过这些工作的训练流程里，预热都还保留着。\n\n### 预热到底有多重要？\n\n作者先在 IWSLT14 德译英的翻译任务上，用 Post-LN Transformer 确认预热的影响。预热的调度是：\n\n$$\n\\mathrm{lr}(t) = \\frac{t}{T_{\\text{warmup}}}\\,\\mathrm{lr}_{\\max}, \\quad t \\le T_{\\text{warmup}}\n$$\n\n之后接逆平方根衰减。他们分别用 Adam 和普通的 SGD 训练，比较有没有预热、预热多久：\n\n[![两组折线图，横轴都是 epoch。(a) 用 Adam 训练：左边是验证损失，右边是 BLEU。没有预热的两条虚线，损失停在 7 左右、BLEU 不到 10；预热 500 步、最大学习率 1e-3 的橙线也很差；预热 4000 步的两条线，损失降到约 4、BLEU 升到 30 以上。(b) 用 SGD 训练：有预热时损失从 10 降到 6 到 7 之间，BLEU 约 7；没有预热时损失停在 8.5 以上，BLEU 接近 0](/content/papers/assets/82e2f8ace58e613addf613e637786b7479429ee4db56d81756a47b8554ff3968.png)](/content/papers/assets/82e2f8ace58e613addf613e637786b7479429ee4db56d81756a47b8554ff3968.png)\n\n*原论文 Figure 2，取自 [**v2 PDF 第 4 页**](https://arxiv.org/pdf/2002.04745v2#page=4)。IWSLT14 德译英的验证损失与 BLEU：(a) 用 Adam，(b) 用 SGD。*\n\n结果很明确：\n\n- **不预热就训练不起来**。用 Adam、不预热，BLEU 只有 8.45；有预热的模型大约是 34。SGD 的成绩本来就比 Adam 差很多，但趋势一样：不预热时，15 个 epoch 里 BLEU 只比 0 高一点点。\n- **预热多久也很敏感**。预热只有 500 步时，最大学习率 5e-4 和 1e-3 的 BLEU 分别只有 31.16 和 2.77。\n\n之前也有人解释过预热的作用。RAdam 的作者认为，预热是在训练初期压低 Adam 自适应学习率的方差。但这里 SGD 同样需要预热，所以作者认为预热的好处和特定的优化器无关，原因要到模型本身去找。\n\n## 解决问题\n\n### 看初始化的那一刻\n\n预热只发生在最开始的几千步，所以作者把焦点放在初始化上：网络刚建好、参数都是随机值的时候，梯度长什么样子？\n\n他们用平均场理论来分析，并做了几个简化：只用单头注意力；所有权重矩阵都是 $d\\times d$，用 Xavier 初始化；注意力的 $W^Q$、$W^K$ 初始化为 0，让注意力一开始是均匀分布；输入向量也当作高斯分布。\n\n分析分三步：先看隐藏层的大小，再看 LayerNorm 怎么处理梯度，最后把两者连起来。\n\n### 第一步：主干上的信号有多大？\n\nPost-LN 每一层最后都做 LayerNorm，所以每层输出的范数固定在 $\\sqrt{d}$ 左右。作者改看最后一次归一化之前的 $x^{\\text{post},5}$，也就是 FFN 输出加上残差之后的向量。Pre-LN 的主干没有归一化，每一层都往上加东西。作者证明了：\n\n$$\n\\mathbb{E}\\big\\|x^{\\text{post},5}_{l}\\big\\|_2^2 = \\tfrac{3}{2}d, \\qquad\n\\Big(1+\\tfrac{l}{2}\\Big)d \\le \\mathbb{E}\\big\\|x^{\\text{pre}}_{l}\\big\\|_2^2 \\le \\Big(1+\\tfrac{3l}{2}\\Big)d\n$$\n\nPost-LN 每一层都一样大；Pre-LN 随着层数线性增长。我们按论文的简化设置跑一遍，看看是不是这样：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\nd, n, L = 512, 8, 12                       # 隐藏维度、序列长度、层数\nln = lambda v: F.layer_norm(v, (d,))\nxavier = lambda: torch.randn(d, d) / d ** 0.5   # Xavier：d×d 的矩阵，方差 2/(d+d) = 1/d\n\ndef attention(h, Wv):                      # 论文的简化设置：W_Q = W_K = 0，注意力是均匀分布\n    return (h @ Wv).mean(dim=0, keepdim=True).expand_as(h)\n\ndef ffn(h, W1, W2):\n    return F.relu(h @ W1) @ W2\n\nx_post = x_pre = torch.randn(n, d)\nfor l in range(1, L + 1):\n    Wv, W1, W2 = xavier(), xavier(), xavier()\n    # Post-LN：相加之后归一化\n    x3 = ln(x_post + attention(x_post, Wv))\n    x5 = x3 + ffn(x3, W1, W2)\n    x_post = ln(x5)\n    # Pre-LN：先归一化再进子层，主干直接相加\n    x_mid = x_pre + attention(ln(x_pre), Wv)\n    x_pre = x_mid + ffn(ln(x_mid), W1, W2)\n    if l in (1, 2, 4, 8, 12):\n        post = (x5 ** 2).sum(1).mean() / d\n        pre = (x_pre ** 2).sum(1).mean() / d\n        print(f\"第 {l:2d} 层后：Post-LN 的 ‖x⁽ᵖᵒˢᵗ,⁵⁾‖²/d = {post:.2f}   \"\n              f\"Pre-LN 的 ‖x‖²/d = {pre:5.2f}（理论范围 {1 + l / 2:.1f}～{1 + 3 * l / 2:.1f}）\")\n```\n\n```text\n第  1 层后：Post-LN 的 ‖x⁽ᵖᵒˢᵗ,⁵⁾‖²/d = 1.51   Pre-LN 的 ‖x‖²/d =  1.62（理论范围 1.5～2.5）\n第  2 层后：Post-LN 的 ‖x⁽ᵖᵒˢᵗ,⁵⁾‖²/d = 1.41   Pre-LN 的 ‖x‖²/d =  2.27（理论范围 2.0～4.0）\n第  4 层后：Post-LN 的 ‖x⁽ᵖᵒˢᵗ,⁵⁾‖²/d = 1.55   Pre-LN 的 ‖x‖²/d =  4.41（理论范围 3.0～7.0）\n第  8 层后：Post-LN 的 ‖x⁽ᵖᵒˢᵗ,⁵⁾‖²/d = 1.54   Pre-LN 的 ‖x‖²/d =  9.73（理论范围 5.0～13.0）\n第 12 层后：Post-LN 的 ‖x⁽ᵖᵒˢᵗ,⁵⁾‖²/d = 1.32   Pre-LN 的 ‖x‖²/d = 15.44（理论范围 7.0～19.0）\n```\n\nPost-LN 一直在 1.5 附近（这只是一次抽样，会有些起伏）；Pre-LN 一层层往上涨，而且都落在理论范围内。\n\n$\\tfrac{3}{2}$ 是怎么来的？$x^{\\text{post},3}$ 刚做完 LayerNorm，平方范数是 $d$。它经过 FFN：乘上 Xavier 初始化的 $W^1$ 之后，每个元素的方差是 1；ReLU 砍掉一半，平方范数剩 $d/2$；再乘上 $W^2$，还是 $d/2$。两者相加，就是 $d + d/2 = \\tfrac{3}{2}d$。Pre-LN 每一层也会加进至少 FFN 的那 $d/2$，这就是下限里的 $l/2$。\n\n### 第二步：LayerNorm 会把梯度除以输入的大小\n\nLayerNorm 把输入减去均值、除以标准差。输入放大 $k$ 倍，输出不变；所以反过来，输出对输入的导数会缩小 $k$ 倍。作者把这件事写成：\n\n$$\n\\big\\|J_{\\mathrm{LN}}(x)\\big\\|_2 = O\\!\\left(\\frac{\\sqrt{d}}{\\|x\\|_2}\\right)\n$$\n\n$J_{\\mathrm{LN}}$ 是 LayerNorm 的 Jacobian 矩阵。直接算一遍：\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom torch.autograd.functional import jacobian\n\ntorch.manual_seed(0)\nd = 256\nz = torch.randn(d)\nfor k in (1, 2, 4, 8):\n    x = k * z                                          # 同一个方向，长度放大 k 倍\n    J = jacobian(lambda v: F.layer_norm(v, (d,)), x)   # LayerNorm 对输入的 Jacobian，形状 [d, d]\n    print(f\"‖x‖ = {x.norm():6.1f}   Jacobian 的谱范数 = {torch.linalg.matrix_norm(J, ord=2):.4f}\"\n          f\"   √d / ‖x‖ = {d ** 0.5 / x.norm():.4f}\")\n```\n\n```text\n‖x‖ =   15.0   Jacobian 的谱范数 = 1.0651   √d / ‖x‖ = 1.0647\n‖x‖ =   30.1   Jacobian 的谱范数 = 0.5325   √d / ‖x‖ = 0.5323\n‖x‖ =   60.1   Jacobian 的谱范数 = 0.2663   √d / ‖x‖ = 0.2662\n‖x‖ =  120.2   Jacobian 的谱范数 = 0.1331   √d / ‖x‖ = 0.1331\n```\n\n输入长度每翻一倍，Jacobian 就减半，和 $\\sqrt{d}/\\|x\\|$ 几乎一模一样。梯度往回传经过 LayerNorm 时，就是乘上这个 Jacobian：输入越大，传回去的梯度越小。\n\n### 第三步：连起来\n\n把前两步连在一起，就是这篇论文的主要定理。作者分析的是最后一层 FFN 的第二个权重矩阵 $W^{2,L}$，结论是（在一个以高概率成立的条件下）：\n\n$$\n\\text{Post-LN：}\\left\\|\\frac{\\partial \\tilde{\\mathcal{L}}}{\\partial W^{2,L}}\\right\\|_F \\le O\\big(d\\sqrt{\\ln d}\\big), \\qquad\n\\text{Pre-LN：}\\left\\|\\frac{\\partial \\tilde{\\mathcal{L}}}{\\partial W^{2,L}}\\right\\|_F \\le O\\left(d\\sqrt{\\frac{\\ln d}{L}}\\right)\n$$\n\n道理是这样的。Post-LN 最后一层 LayerNorm 的输入是 $x^{\\text{post},5}$，大小固定在 $\\sqrt{3d/2}$ 左右，和层数无关，所以最后一层的梯度也和层数无关。Pre-LN 最后那层 LayerNorm 的输入是一路累加上来的主干，平方范数随层数 $L$ 线性增长，范数大约是 $\\sqrt{L}$ 的量级；梯度经过它，就被除以 $\\sqrt{L}$。\n\n作者还做了一些初步的推广分析，结论是：Post-LN 越靠近输出层的参数，梯度越大，越往输入端越小；Pre-LN 各层的梯度大致相同。\n\n### 实际量一次\n\n理论做了不少简化，作者用真实的模型做了检验。他们先确认隐藏层范数的集中性质成立，再测量梯度：\n\n[![四张图。(a)(b) 是 6 层编码器、6 层解码器的 Transformer 中，每一层 FFN 的 W1 与 W2 的梯度大小柱状图：蓝色 Pre-LN（初始化时）各层的高度变化不大；橙色 Post-LN（初始化时）随层数递增，第 6 层最高；绿色 Post-LN（预热之后）几乎是 0。(c) Pre-LN 最后一层 W2 的梯度随总层数从 6 增加到 14 而下降，从 0.3 降到约 0.13。(d) Post-LN 最后一层 W2 的梯度保持在 1.6 左右，不随层数改变](/content/papers/assets/1365260baf3a0bcbf5df3b21d3b56df47b2abd8ebd257a7c0472af1a24dd360a.png)](/content/papers/assets/1365260baf3a0bcbf5df3b21d3b56df47b2abd8ebd257a7c0472af1a24dd360a.png)\n\n*原论文 Figure 3，取自 [**v2 PDF 第 7 页**](https://arxiv.org/pdf/2002.04745v2#page=7)。(a)(b) 6-6 层 Transformer 各层 FFN 权重的梯度大小；(c)(d) 不同大小的模型中，最后一层 $W^2$ 的梯度大小。每组实验用十个随机种子重复。*\n\n和理论对得上：\n\n- 右边两张：Post-LN 最后一层的梯度不管模型多大，都在 1.6 左右；Pre-LN 随着层数变多而下降。\n- 左边两张：Post-LN 的梯度从第 1 层到第 6 层一路变大，最靠近输出的那层最大；Pre-LN 各层的差异小得多。\n\n所以 Post-LN 在初始化时，靠近输出的参数梯度很大。这时直接用大的学习率，更新步子就会迈得太大，训练不稳定。预热正好避开了这一段：等梯度变小了，才把学习率调上去。图上绿色的柱子（Post-LN 预热之后）几乎都是 0，梯度确实已经小了很多。\n\n作者另外试了不预热、全程用固定的小学习率 1e-4 训练 Post-LN。这样也能把模型训练到一定程度，但收敛慢得多。\n\n反过来看，Pre-LN 在初始化时梯度就很规矩，那它应该可以去掉预热。\n\n## 讨论\n\n### 翻译：不预热，还更快\n\n作者在两个翻译任务上做了验证：\n\n- **IWSLT14 德译英**：6 层编码器、6 层解码器，嵌入维度 512，FFN 1024，4 个注意力头。\n- **WMT14 英译德**：Transformer base 的设置，约 450 万组平行句对。\n\nPost-LN 照原始论文预热 4000 步，再接逆平方根衰减。Pre-LN 完全不预热：IWSLT14 从 5e-4 开始，第 8 个 epoch 乘以 0.1；WMT14 分别从 7e-4 和 1.5e-3 开始，第 6 个 epoch 衰减，之后接逆平方根衰减。两者的最大学习率设成一样。RAdam 号称可以去掉预热，作者也在 IWSLT14 上一起做了比较。\n\n[![四张折线图，横轴都是 epoch。(a)(b) 是 IWSLT14 的验证损失与 BLEU：Pre-LN（不论 Adam 还是 RAdam，都不预热）的红线与橙线在前几个 epoch 就降到很低、BLEU 很快升到 30 以上；Post-LN 用 Adam 加预热的绿线，以及用 RAdam 不预热的蓝线，都爬得比较慢，最后才追上。(c)(d) 是 WMT14 的验证损失与 BLEU：两种学习率的 Pre-LN 都比对应的 Post-LN 更早收敛，最后 BLEU 都在 26 左右](/content/papers/assets/26ae58e24cef2c40c73b93b2c17d74d07d2a6fafe71d071e70ed0a490d9d585f.png)](/content/papers/assets/26ae58e24cef2c40c73b93b2c17d74d07d2a6fafe71d071e70ed0a490d9d585f.png)\n\n*原论文 Figure 4，取自 [**v2 PDF 第 8 页**](https://arxiv.org/pdf/2002.04745v2#page=8)。IWSLT14 德译英（a、b）与 WMT14 英译德（c、d）的验证损失与 BLEU。*\n\n三个观察：\n\n1. **预热不再必要**。在 IWSLT14 上，不预热的 Pre-LN 最后 BLEU 约 34、验证损失约 4，和预热的 Post-LN 相当。\n2. **收敛更快**。在 IWSLT14 上，Pre-LN 第 9 个 epoch 的表现，和 Post-LN 第 15 个 epoch 差不多；WMT14 也有类似的情况。\n3. **LayerNorm 的位置比优化器更重要**。RAdam 确实能让 Post-LN 不预热也训练得起来；可是换成 Pre-LN 之后，用 RAdam 还是 Adam 几乎没有区别。\n\n作者也提到，如果把超参数调好（初始学习率 7.5e-4、dropout 0.3 等），Pre-LN 在 IWSLT14 上可以做到 35.5 的测试 BLEU。\n\n### BERT：省下四成\n\n预训练用的是 BERT base（12 层、隐藏维度 768、12 个头），数据是英文维基百科加上作者自己收集的 BookCorpus，总共约 34 亿词。\n\nPost-LN BERT 照原论文的设置：预热 10,000 步，最大学习率 1e-4。作者试过把 Post-LN 的学习率提高到 3e-4，训练直接发散。Pre-LN BERT 则不预热，从 3e-4 开始线性衰减。\n\n[![三张折线图，横轴都是预训练步数（千步）。(a) BERT 的验证损失：橙色的 Pre-LN 从一开始就低于蓝色虚线的 Post-LN，两条线大致平行下降，Pre-LN 约在 500 千步降到 1.69，Post-LN 要到约 700 千步。(b) MRPC 准确率与 (c) RTE 准确率：各个预训练阶段的 Pre-LN 都高于 Post-LN，差距在预训练初期最明显](/content/papers/assets/9707b62474bc0eba4a5ff16f9578eb437856e931b5127793ec370b6a1b0d0a06.png)](/content/papers/assets/9707b62474bc0eba4a5ff16f9578eb437856e931b5127793ec370b6a1b0d0a06.png)\n\n*原论文 Figure 5，取自 [**v2 PDF 第 8 页**](https://arxiv.org/pdf/2002.04745v2#page=8)。(a) BERT 预训练的验证损失；(b)(c) 不同预训练阶段的模型在 MRPC、RTE 上微调后的准确率，误差线是六次微调的 95% 置信区间。*\n\nPost-LN 要训练到第 700k 步，验证损失才降到 1.69；Pre-LN 第 500k 步就到了，大约快了 40%。省下的 200k 步，远多于预热本身的 10k 步，所以快的原因不只是少了预热，更在于 Pre-LN 撑得住大得多的学习率。\n\n拿不同阶段的模型去微调 MRPC 和 RTE，Pre-LN 在下游任务上也收敛得更快。\n\n## 结论\n\n这篇论文把一个工程上的习惯，追溯到了架构上的原因：Post-LN 在初始化时，靠近输出的参数梯度很大，必须用预热来避开；把 LayerNorm 搬进残差分支、放在子层之前，最后一层的梯度会被主干的大小按 $1/\\sqrt{L}$ 缩小，各层的梯度也更均匀，预热就可以去掉，学习率还能开得更大。\n\n要记住理论分析的范围：它看的是初始化的那一刻，又做了均匀注意力、单头这些简化；「Post-LN 越靠近输出梯度越大」那部分，作者自己也标成了初步结果，主要靠 Figure 3 的实测支持。\n\n「把归一化搬进分支、让捷径保持干净」的想法，在卷积网络上也出现过；GPT-2 也把 LayerNorm 移到了每个子层的输入处：\n\n\n> [!TIP]\n> \n> - **[[16.03] ResNet v2：捷径要干净](/blog/papers/classic-cnns/resnet-v2)**\n> - **[[19.02] GPT-2：四十八层解码器](/blog/papers/transformers/gpt_2)**\n\n\n如果你要从头训练 Transformer，这篇的建议很直接：LayerNorm 放在子层前面，输出前记得再补一层 LayerNorm。在论文的实验规模里，这样不预热也训练得起来，学习率还能开大一点。\n\nPre-LN 后来成了很多大语言模型的默认摆法，但它也有代价。从第一步的结果就看得出来：主干的平方范数随层数线性增长，每一层加进去的量却差不多，越往后的层，对主干的相对影响就越小。2021 年的 [NormFormer](https://arxiv.org/abs/2110.09456) 指出，Pre-LN 预训练时前面几层的梯度比后面大得多，于是在每层多加了几个归一化；2022 年的 [DeepNet](https://arxiv.org/abs/2203.00555) 则回头改造 Post-LN，想同时保留 Post-LN 的表现和 Pre-LN 的稳定，把 Transformer 推到了一千层。\n\nLayerNorm 到底该放在哪里，这个话题还没结束，之后有机会我们再来聊。\n","imageSizes":{"/content/papers/assets/f44d59bcadead95b9b93a42fc681bc473b038c13ed0c5c9e1effc7ee90409fc3.png":{"width":436,"height":531},"/content/papers/assets/82e2f8ace58e613addf613e637786b7479429ee4db56d81756a47b8554ff3968.png":{"width":1348,"height":391},"/content/papers/assets/1365260baf3a0bcbf5df3b21d3b56df47b2abd8ebd257a7c0472af1a24dd360a.png":{"width":1332,"height":263},"/content/papers/assets/26ae58e24cef2c40c73b93b2c17d74d07d2a6fafe71d071e70ed0a490d9d585f.png":{"width":1342,"height":276},"/content/papers/assets/9707b62474bc0eba4a5ff16f9578eb437856e931b5127793ec370b6a1b0d0a06.png":{"width":1011,"height":369}}}