{"notes":[{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"捷径要干净","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnet-v2.9e686e0339ca.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"两个恒等映射","text":"两个恒等映射","depth":3},{"id":"捷径上多一点东西会怎样","text":"捷径上多一点东西会怎样？","depth":3},{"id":"相加之后的-relu-该放在哪","text":"相加之后的 ReLU 该放在哪？","depth":3},{"id":"换个角度看pre-activation","text":"换个角度看：pre-activation","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"越深差距越大","text":"越深，差距越大","depth":3},{"id":"和其他方法比较","text":"和其他方法比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"卷积网络 100 层","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnet.98d0474f71e6.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"什么是残差","text":"什么是残差？","depth":3},{"id":"残差学习","text":"残差学习","depth":3},{"id":"残差模块","text":"残差模块","depth":3},{"id":"网络架构","text":"网络架构","depth":3},{"id":"参数配置","text":"参数配置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"更深的瓶颈架构","text":"更深的瓶颈架构","depth":3},{"id":"在-imagenet-上的表现","text":"在 ImageNet 上的表现","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"如何训练更深的卷积网络"},{"id":"classic-cnns/densenet","relation":"next","topic":"如何训练更深的卷积网络"}]},{"id":"classic-cnns/highway-networks","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Highway Networks","subtitle":"残差的前夜","author":"Z. Yuan","paperDate":"2015-05-03","publishedOn":"2025-11-03","world":"external/highway-networks","paperTitle":"Highway Networks","paperUrl":"https://arxiv.org/abs/1505.00387","bibliography":{"title":"Highway Networks","authors":["Rupesh Kumar Srivastava","Klaus Greff","Jürgen Schmidhuber"],"url":"https://arxiv.org/abs/1505.00387","pdfUrl":"https://arxiv.org/pdf/1505.00387v2","codeUrl":null,"preprint":{"id":"1505.00387","submittedOn":"2015-05-03","url":"https://arxiv.org/abs/1505.00387"},"publication":null,"citation":"@misc{classiccnnshighwaynetworks2015,\n  title = {{Highway Networks}},\n  author = {Rupesh Kumar Srivastava and Klaus Greff and Jürgen Schmidhuber},\n  year = {2015},\n  eprint = {1505.00387},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1505.00387}\n}\n","citationPath":"/content/papers/citations/classic-cnns/highway-networks.f7d3ef9547ef.bib"},"thumbnail":{"src":"/content/papers/assets/2cef37bd2a2388f57a76965431810cb86f976db701ba1647b75c81c9a59ac779.png","width":1344,"height":360},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/highway-networks.5d61f447ab06.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"加两道门","text":"加两道门","depth":3},{"id":"一开始先偏向直接通过","text":"一开始先偏向「直接通过」","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"越深越好训练吗","text":"越深越好训练吗？","depth":3},{"id":"好训练那准确率呢","text":"好训练，那准确率呢？","depth":3},{"id":"门学到了什么","text":"门学到了什么？","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/inception-v4","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v4","subtitle":"快很多，准一点","author":"Z. Yuan","paperDate":"2016-02-23","publishedOn":"2026-01-27","world":"external/inception-v4","paperTitle":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","paperUrl":"https://arxiv.org/abs/1602.07261","bibliography":{"title":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","authors":["Christian Szegedy","Sergey Ioffe","Vincent Vanhoucke","Alex Alemi"],"url":"https://arxiv.org/abs/1602.07261","pdfUrl":"https://arxiv.org/pdf/1602.07261v2","codeUrl":null,"preprint":{"id":"1602.07261","submittedOn":"2016-02-23","url":"https://arxiv.org/abs/1602.07261"},"publication":null,"citation":"@misc{classiccnnsinceptionv42016,\n  title = {{Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning}},\n  author = {Christian Szegedy and Sergey Ioffe and Vincent Vanhoucke and Alex Alemi},\n  year = {2016},\n  eprint = {1602.07261},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1602.07261}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v4.ed8e0294ae0b.bib"},"thumbnail":{"src":"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png","width":666,"height":656},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inception-v4.615919e732a6.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"卸下包袱inception-v4","text":"卸下包袱：Inception-v4","depth":3},{"id":"进场的那一段","text":"进场的那一段","depth":3},{"id":"把残差接进-inception","text":"把残差接进 Inception","depth":3},{"id":"缩小残差稳住训练","text":"缩小残差，稳住训练","depth":3},{"id":"训练设置","text":"训练设置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"残差让训练快了很多","text":"残差让训练快了很多","depth":3},{"id":"单个模型的成绩","text":"单个模型的成绩","depth":3},{"id":"集成","text":"集成","depth":3},{"id":"残差连接是必需的吗","text":"残差连接是「必需」的吗？","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""}]}],"text":"ResNet 让 152 层的网络训练得动，拿下了 ILSVRC 2015。可是在 CIFAR-10 上，原论文的 1202 层 ResNet 错误率是 7.93%，反而比 110 层的 6.61% 差。\n\n残差连接能让网络变深，可是深到一千层，又出现了新的问题。\n\n2016 年 3 月，原班人马 Kaiming He、Xiangyu Zhang、Shaoqing Ren 与 Jian Sun 发布了这篇论文，后来发表在 ECCV 2016。这次他们回头写出了残差单元的传播公式，从公式里找出信号怎么走才最顺畅，再照着改造残差单元。改完之后，1001 层的 ResNet 在 CIFAR-10 上做到了 4.62% 的错误率。\n\n这个新版本，后来常被叫做 ResNet v2，或 pre-activation ResNet。\n\n\n> [!TIP]\n> \n> ResNet 原本的设计，可以先看我们的笔记：\n> \n> - **[[15.12] ResNet：卷积网络 100 层](/blog/papers/classic-cnns/resnet)**\n\n\n## 定义问题\n\n先把原本的残差单元写成一般形式。第 $l$ 个单元的输入是 $x_l$：\n\n$$\ny_l = h(x_l) + \\mathcal{F}(x_l, \\mathcal{W}_l), \\qquad x_{l+1} = f(y_l)\n$$\n\n$\\mathcal{F}$ 是残差函数，例如两层 $3\\times3$ 卷积；$h$ 是捷径上的运算；$f$ 是相加之后的运算。在原本的 ResNet 里，$h(x_l) = x_l$ 是恒等映射，$f$ 是 ReLU。\n\n[![左边是两个残差单元的示意图。(a) original：x_l 一路往下，右侧分支依次是 weight、BN、ReLU、weight、BN，和捷径相加后再经过 ReLU 得到 x_l+1。(b) proposed：右侧分支依次是 BN、ReLU、weight、BN、ReLU、weight，相加之后直接得到 x_l+1。两者的捷径都画成一条灰色的粗箭头。右边是 CIFAR-10 上 1001 层 ResNet 的训练曲线：黄色是 original（错误率 7.61%），蓝色是 proposed（错误率 4.92%）；虚线是训练损失，实线是测试错误率，蓝线的训练损失一开始就降得快很多](/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png)](/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png)\n\n*原论文 Figure 1，取自 [**v3 PDF 第 2 页**](https://arxiv.org/pdf/1603.05027v3#page=2)。左：(a) 原本的残差单元，(b) 本文提出的残差单元；灰色箭头是信息最容易传播的路径。右：CIFAR-10 上 1001 层 ResNet 的训练曲线，虚线是训练损失（左轴），实线是测试错误率（右轴）。*\n\n右边的曲线把问题摆在了眼前：同样是 1001 层，原本的设计在训练初期损失降得很慢，最后的测试错误率是 7.61%；改过的设计一开始就降得很快，最后是 4.92%。\n\n左边两个单元用的组件完全一样，区别只在顺序：原本的 ReLU 放在相加之后，压在捷径上；新的设计把 BN 和 ReLU 都搬到权重层前面，相加之后什么都不做。灰色的捷径因此从头到尾都没有经过任何运算。\n\n为什么这件事重要？作者的论证，要从把 $h$ 和 $f$ 都当作恒等映射开始。\n\n## 解决问题\n\n### 两个恒等映射\n\n如果 $f$ 也是恒等映射，$x_{l+1} = y_l$，上面的式子就变成：\n\n$$\nx_{l+1} = x_l + \\mathcal{F}(x_l, \\mathcal{W}_l)\n$$\n\n一路展开到更深的第 $L$ 个单元：\n\n$$\nx_L = x_l + \\sum_{i=l}^{L-1} \\mathcal{F}(x_i, \\mathcal{W}_i)\n$$\n\n这个式子说的是：任何一个深层的特征 $x_L$，都等于某个浅层的特征 $x_l$，加上中间所有残差函数的输出。对照普通网络，$x_L$ 是一连串矩阵乘法的结果，大约是 $\\prod_i W_i\\, x_0$（忽略 BN 和 ReLU）。一个是加起来，一个是乘起来。\n\n反向传播时也一样。设损失为 $\\mathcal{E}$，由链式法则：\n\n$$\n\\frac{\\partial \\mathcal{E}}{\\partial x_l}\n= \\frac{\\partial \\mathcal{E}}{\\partial x_L}\\frac{\\partial x_L}{\\partial x_l}\n= \\frac{\\partial \\mathcal{E}}{\\partial x_L}\\left(1 + \\frac{\\partial}{\\partial x_l}\\sum_{i=l}^{L-1}\\mathcal{F}(x_i, \\mathcal{W}_i)\\right)\n$$\n\n括号里的 1，让 $\\partial\\mathcal{E}/\\partial x_L$ 不经过任何权重层，直接传到浅层的 $x_l$。另一项要经过权重层，但它不太可能对一个批量里的每个样本都恰好是 $-1$，把整个括号抵消掉。所以即使权重很小，梯度也不会消失。\n\n这两个漂亮的性质，都建立在两个条件上：捷径 $h$ 是恒等映射，相加之后的 $f$ 也是恒等映射。换成图上的说法，就是灰色箭头上除了相加，不能有任何运算。\n\n作者接下来分两部分检验：先改动捷径 $h$，再改动相加之后的 $f$。\n\n### 捷径上多一点东西会怎样？\n\n最简单的破坏方式，是让捷径乘上一个常数 $\\lambda_l$：$h(x_l) = \\lambda_l x_l$。用同样的方法展开，反向传播的式子变成：\n\n$$\n\\frac{\\partial \\mathcal{E}}{\\partial x_l}\n= \\frac{\\partial \\mathcal{E}}{\\partial x_L}\\left(\\prod_{i=l}^{L-1}\\lambda_i + \\frac{\\partial}{\\partial x_l}\\sum_{i=l}^{L-1}\\hat{\\mathcal{F}}(x_i, \\mathcal{W}_i)\\right)\n$$\n\n$\\hat{\\mathcal{F}}$ 把那些常数吸收了进去。原本的 1，变成了一串 $\\lambda$ 的连乘。网络很深时，只要每个 $\\lambda$ 都大于 1，这一项就会指数级爆炸；都小于 1，它就会指数级消失，梯度只能改走权重层。\n\nResNet-110 有 54 个残差单元。我们把残差函数设得很小，只看捷径的效果：\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nD, units = 64, 54                                       # ResNet-110 有 54 个残差单元\nWs = [torch.randn(D, D) * 0.01 for _ in range(units)]   # 刻意让残差函数很小\n\nfor lam in (1.0, 0.9, 0.5, 1.1):\n    x0 = torch.randn(D, requires_grad=True)\n    x = x0\n    for W in Ws:\n        x = lam * x + torch.relu(x) @ W                 # 捷径乘上 λ\n    x.sum().backward()\n    print(f\"λ = {lam}：传回第一层的梯度约 {x0.grad.abs().mean().item():.3g}，λ^54 = {lam ** units:.3g}\")\n```\n\n```text\nλ = 1.0：传回第一层的梯度约 0.954，λ^54 = 1\nλ = 0.9：传回第一层的梯度约 0.00322，λ^54 = 0.00338\nλ = 0.5：传回第一层的梯度约 5.67e-17，λ^54 = 5.55e-17\nλ = 1.1：传回第一层的梯度约 162，λ^54 = 172\n```\n\n梯度几乎就是 $\\lambda^{54}$。$\\lambda = 0.9$ 看起来只打了九折，经过 54 个单元之后只剩千分之三；$\\lambda = 0.5$ 则小到 $10^{-17}$。换成门控或 $1\\times1$ 卷积这类更复杂的捷径，连乘的就变成每个捷径的导数，同样可能卡住信号。\n\n作者接着在 CIFAR-10 的 ResNet-110 上实际比较了几种捷径：\n\n[![六个残差单元的示意图，右侧分支都是 3x3 conv、ReLU、3x3 conv，区别在捷径上：(a) original 没有任何运算；(b) constant scaling 在捷径和分支上各乘 0.5；(c) exclusive gating 用 1x1 conv 加 sigmoid 算出门控值，分支乘上门控值、捷径乘上 1 减门控值；(d) shortcut-only gating 只在捷径上乘 1 减门控值；(e) conv shortcut 捷径上是 1x1 conv；(f) dropout shortcut 捷径上是 dropout。每个单元相加之后都接 ReLU](/content/papers/assets/7710a267ec379977c4b41b07fbb2532dd44111584b6c0542b7ce9e96556e4f13.png)](/content/papers/assets/7710a267ec379977c4b41b07fbb2532dd44111584b6c0542b7ce9e96556e4f13.png)\n\n*原论文 Figure 2，取自 [**v3 PDF 第 5 页**](https://arxiv.org/pdf/1603.05027v3#page=5)。Table 1 用到的各种捷径。为了简化，图上没有画出每个权重层后面的 BN。*\n\n| 捷径 | 捷径上的系数 | 分支 $\\mathcal{F}$ 上的系数 | 错误率 |\n| --- | --- | --- | ---: |\n| (a) 原本的恒等捷径 | 1 | 1 | **6.61%** |\n| (b) 常数缩放 | 0 | 1 | 失败（相当于普通网络） |\n| (b) 常数缩放 | 0.5 | 1 | 失败 |\n| (b) 常数缩放 | 0.5 | 0.5 | 12.35% |\n| (c) 互斥门控，$b_g$ 初始为 $-6$ | $1-g(x)$ | $g(x)$ | 8.70% |\n| (c) 互斥门控，$b_g$ 初始为 0 到 $-5$ | $1-g(x)$ | $g(x)$ | 失败 |\n| (d) 只在捷径上加门控，$b_g$ 初始为 0 | $1-g(x)$ | 1 | 12.86% |\n| (d) 只在捷径上加门控，$b_g$ 初始为 $-6$ | $1-g(x)$ | 1 | 6.91% |\n| (e) $1\\times1$ 卷积捷径 | $1\\times1$ 卷积 | 1 | 12.22% |\n| (f) dropout 捷径 | dropout 0.5 | 1 | 失败 |\n\n*表：摘自原论文 Table 1 的部分行。CIFAR-10 测试集，ResNet-110，五次训练的中位数；错误率超过 20% 记为「失败」。这组实验在相加之后仍用 ReLU。*\n\n几个值得注意的地方：\n\n- **常数缩放 0.5**：分支不缩放时训练不起来；分支也乘 0.5（类似门控固定住的 Highway），可以收敛，但错误率 12.35%，接近基线的两倍。\n- **互斥门控**就是 Highway Networks 的做法。门控偏置的初始值非常关键，作者在 0 到 $-10$ 之间搜索，最好的 $-6$ 也只做到 8.70%。\n- **只在捷径上加门控**时，偏置初始为 $-6$ 会让 $1-g(x)$ 接近 1，捷径几乎就是恒等映射，结果 6.91% 也最接近基线。\n- **$1\\times1$ 卷积捷径**在原本 34 层的 ResNet 上表现不错（ResNet 论文里的选项 C），堆到 54 个单元却变成了 12.22%。作者在 ImageNet 的 ResNet-101 上也看到了类似的现象。\n- **dropout 捷径**在统计上相当于把捷径乘上期望值 0.5，和常数缩放一样会卡住信号。\n\n\n> [!TIP]\n> \n> 互斥门控的原型，可以看我们的笔记：\n> \n> - **[[15.05] Highway Networks：残差的前夜](/blog/papers/classic-cnns/highway-networks)**\n\n\n门控和 $1\\times1$ 卷积都多了参数，理论上表达能力应该更强；只在捷径上加门控、或者用 $1\\times1$ 卷积时，解空间仍然包含恒等捷径，只要学成恒等映射就行。可是它们的**训练**错误率都更高。所以变差的原因出在优化上，与表达能力无关。\n\n### 相加之后的 ReLU 该放在哪？\n\n上一组实验里，相加之后的 $f$ 都还是 ReLU，所以推导只是近似成立。接下来要处理 $f$。\n\n要让 $f$ 变成恒等映射，只能重新排列 ReLU 和 BN 的位置。作者试了这几种排法：\n\n[![五个残差单元的示意图，组件相同、顺序不同。(a) original：weight、BN、ReLU、weight、BN，相加后 ReLU。(b) BN after addition：weight、BN、ReLU、weight，相加后 BN 再 ReLU。(c) ReLU before addition：weight、BN、ReLU、weight、BN、ReLU，再相加。(d) ReLU-only pre-activation：ReLU、weight、BN、ReLU、weight、BN，再相加。(e) full pre-activation：BN、ReLU、weight、BN、ReLU、weight，再相加](/content/papers/assets/f032506b6dda998779a8abf2fee9cbf44371ac4a2c4ebf8c93d834286bbb7501.png)](/content/papers/assets/f032506b6dda998779a8abf2fee9cbf44371ac4a2c4ebf8c93d834286bbb7501.png)\n\n*原论文 Figure 4，取自 [**v3 PDF 第 8 页**](https://arxiv.org/pdf/1603.05027v3#page=8)。Table 2 用到的各种激活位置；所有单元的组件都一样，只有顺序不同。*\n\n| 排法 | ResNet-110 | ResNet-164 |\n| --- | ---: | ---: |\n| (a) 原本的残差单元 | 6.61% | 5.93% |\n| (b) BN 放在相加之后 | 8.17% | 6.50% |\n| (c) ReLU 放在相加之前 | 7.84% | 6.14% |\n| (d) 只把 ReLU 前移 | 6.71% | 5.91% |\n| (e) 完整的 pre-activation | **6.37%** | **5.46%** |\n\n*表：摘自原论文 Table 2。CIFAR-10 测试集错误率。ResNet-164 用的是瓶颈结构（$1\\times1$、$3\\times3$、$1\\times1$），计算量和两层 $3\\times3$ 的单元相近。*\n\n**(b) BN 放在相加之后**是反方向的实验：让 $f$ 包含 BN 和 ReLU。结果明显变差。BN 这时会改变捷径上的信号，训练初期的损失很难降下来。\n\n**(c) ReLU 放在相加之前**是让 $f$ 变成恒等映射最直接的做法，结果也变差了。问题在于，ReLU 放在分支的最后，残差函数的输出就永远不会是负的。直觉上，「残差」应该可正可负；只能是正的，信号就会一路单调增加。我们看看两种排法叠十个单元之后的样子：\n\n```python\nimport torch\nimport torch.nn as nn\n\ntorch.manual_seed(0)\nc = 16\n\ndef relu_before_add():          # Fig. 4(c)：ReLU 移到相加之前\n    return nn.Sequential(nn.Conv2d(c, c, 3, padding=1), nn.BatchNorm2d(c), nn.ReLU(),\n                         nn.Conv2d(c, c, 3, padding=1), nn.BatchNorm2d(c), nn.ReLU())\n\ndef full_preact():              # Fig. 4(e)：BN、ReLU 都放在权重层前面\n    return nn.Sequential(nn.BatchNorm2d(c), nn.ReLU(), nn.Conv2d(c, c, 3, padding=1),\n                         nn.BatchNorm2d(c), nn.ReLU(), nn.Conv2d(c, c, 3, padding=1))\n\nx0 = torch.randn(8, c, 16, 16)\nfor name, make in [(\"ReLU 在相加前\", relu_before_add), (\"完整 pre-activation\", full_preact)]:\n    x, lows, means = x0, [], []\n    for unit in range(10):\n        r = make()(x)\n        lows.append(r.min().item())\n        x = x + r\n        means.append(x.mean().item())\n    print(f\"{name}：残差的最小值 {min(lows):.2f}；输出平均值 {means[0]:.2f} → {means[4]:.2f} → {means[9]:.2f}\")\n```\n\n```text\nReLU 在相加前：残差的最小值 0.00；输出平均值 0.38 → 1.97 → 3.96\n完整 pre-activation：残差的最小值 -2.33；输出平均值 -0.01 → 0.07 → 0.09\n```\n\nReLU 放在相加前，残差最小就是 0，每加一个单元，平均值就往上垫高一截；pre-activation 的残差有正有负，平均值停在 0 附近。\n\n### 换个角度看：pre-activation\n\n剩下的两种排法，来自一个观念上的转换。\n\n在原本的设计里，相加之后的激活 $f$ 会同时影响下一个单元的两条路：\n\n$$\ny_{l+1} = f(y_l) + \\mathcal{F}(f(y_l), \\mathcal{W}_{l+1})\n$$\n\n如果让这个激活只作用在残差分支上，捷径不经过它：\n\n$$\ny_{l+1} = y_l + \\mathcal{F}(\\hat{f}(y_l), \\mathcal{W}_{l+1})\n$$\n\n换个符号，就是 $x_{l+1} = x_l + \\mathcal{F}(\\hat{f}(x_l), \\mathcal{W}_l)$。相加之后什么都没有，$f$ 变成了恒等映射；原本「上一个单元的输出激活」，变成了「下一个单元的输入激活」。\n\n[![三栏示意图。(a) 原本的残差单元，相加之后有 act.，这个 act. 同时影响下一个单元的捷径与分支。(b) 把相加之后的 act. 只放在通往权重的分支上（用红框标出），捷径绕过它。(c) 重新划分单元的边界之后，等价于每个单元在 weight 之前先做 act. 的 pre-activation 残差单元，相加之后没有任何运算](/content/papers/assets/1a7a520a803ed65de475f2ee4bc5ade9b1833de9aefd758c5ef57153e5d087d2.png)](/content/papers/assets/1a7a520a803ed65de475f2ee4bc5ade9b1833de9aefd758c5ef57153e5d087d2.png)\n\n*原论文 Figure 5，取自 [**v3 PDF 第 10 页**](https://arxiv.org/pdf/1603.05027v3#page=10)。把相加之后的激活只用在权重分支上，等价于构建一个 pre-activation 残差单元。*\n\n图上从 (b) 到 (c) 只是把单元的边界重新划分了一次。作者也指出，这种前后之分只在有相加的分支结构里才有意义；一个普通的 $N$ 层网络有 $N-1$ 个激活，叫它预激活还是后激活都一样。\n\n按照这个想法，Figure 4 的 (d) 只把 ReLU 移到前面，结果和原本的设计差不多；作者推测，这个 ReLU 没有和 BN 搭配在一起，享受不到 BN 的好处。(e) 把 BN 和 ReLU 都移到权重层前面，成绩明显变好。这就是本文提出的单元。\n\n## 讨论\n\n### 越深，差距越大\n\n| 数据集 | 网络 | 原本的单元 | pre-activation |\n| --- | --- | ---: | ---: |\n| CIFAR-10 | ResNet-110（每个单元一层） | 9.90% | 8.91% |\n| CIFAR-10 | ResNet-110 | 6.61% | 6.37% |\n| CIFAR-10 | ResNet-164 | 5.93% | 5.46% |\n| CIFAR-10 | ResNet-1001 | 7.61% | **4.92%** |\n| CIFAR-100 | ResNet-164 | 25.16% | 24.33% |\n| CIFAR-100 | ResNet-1001 | 27.82% | **22.71%** |\n\n*表：摘自原论文 Table 3。ResNet-1001 是瓶颈结构，共 333 个残差单元，每种特征图大小各 111 个。*\n\npre-activation 在每一组都更好，而且网络越深，差距越大：1001 层时，CIFAR-10 从 7.61% 降到 4.92%，CIFAR-100 从 27.82% 降到 22.71%。\n\n作者认为好处来自两个方面。\n\n**第一，更容易优化**。这在 1001 层时最明显（Figure 1 右）。$f$ 是 ReLU 时，负的信号会被截断；单元很多时，前面的公式就不再是好的近似。164 层的网络受到的影响小得多：训练初期曲线稍微受挫，很快就恢复了。作者监测了网络的响应，发现训练一段时间之后，权重会调整到让 $y_l$ 大多大于 0，ReLU 很少截断什么；$x_l$ 经过上一个 ReLU 本来就是非负的，只有当 $\\mathcal{F}$ 是很大的负值时，$y_l$ 才会小于 0。到了一千层，截断就频繁得多。\n\n**第二，减少过拟合**。pre-activation 版本收敛时的训练损失略高，测试错误率却更低。作者推测这是 BN 的正则化效果：在原本的单元里，经过 BN 归一化的信号马上就和捷径相加，送进下一个权重层的其实是没有归一化的信号；在 pre-activation 版本里，每个权重层的输入都先经过 BN。\n\n### 和其他方法比较\n\n在 CIFAR 上，作者的 ResNet-1001 在 CIFAR-10 上做到 4.92%（五次中位数），批量大小改成 64 时是 4.62%；CIFAR-100 是 22.71%。他们特别说明，这些结果没有特意调整宽度或滤波器大小，也没有用 dropout 这类对小数据集很有效的正则化，靠的只有「更深」。\n\n在 ImageNet 上，作者先注意到一个评估细节：原本的 ResNet 训练时把短边缩放到 256 到 480 之间，测试时却在短边 256 的图像上取 $224\\times224$ 的裁剪，这对 ResNet 不利。所以这次改成在短边 320 的图像上取一个 $320\\times320$ 的裁剪；ResNet 是全卷积的，可以直接接收更大的输入，这个大小也接近 Inception-v3 的 299。\n\n| 方法 | 数据增强 | Top-1 错误率 | Top-5 错误率 |\n| --- | --- | ---: | ---: |\n| ResNet-152，原本的单元 | 缩放 | 21.3% | 5.5% |\n| ResNet-152，pre-activation | 缩放 | 21.1% | 5.5% |\n| ResNet-200，原本的单元 | 缩放 | 21.8% | 6.0% |\n| ResNet-200，pre-activation | 缩放 | 20.7% | 5.3% |\n| ResNet-200，pre-activation | 缩放＋长宽比 | **20.1%** | **4.8%** |\n| Inception-v3 | 缩放＋长宽比 | 21.2% | 5.6% |\n\n*表：摘自原论文 Table 5。ILSVRC 2012 验证集，单次裁剪；ResNet 以 $224\\times224$ 训练、$320\\times320$ 测试，Inception-v3 训练与测试都是 $299\\times299$。*\n\nResNet-152 换成 pre-activation，只从 21.3% 进步到 21.1%，因为它本来就没有明显的泛化问题。ResNet-200 就不一样了：原本的单元错误率 21.8%，比 ResNet-152 还差，可是它的训练错误率比 ResNet-152 低，也就是过拟合了。换成 pre-activation 之后是 20.7%，比两个 ResNet-152 都好。\n\n作者也在 ImageNet 的 ResNet-101 上跑完了「BN 放在相加之后」的版本：单次裁剪错误率 24.6%／7.5%，比原本的 23.6%／7.1% 差，和 CIFAR 上的结果一致。\n\n\n> [!TIP]\n> \n> 作者在文中也提到了同一时期的 Inception-ResNet-v2，并预期这些发现对它同样有帮助：\n> \n> - **[[16.02] Inception-v4：快很多，准一点](/blog/papers/classic-cnns/inception-v4)**\n\n\n计算量方面，模型的复杂度和深度成正比，1001 层大约是 100 层的十倍。CIFAR 上的 ResNet-1001 用两张 GPU 训练约 27 小时；ImageNet 上的 ResNet-200 用八张 GPU 训练约三周。\n\n## 结论\n\n这篇论文的结论可以浓缩成一句话：让捷径保持干净。捷径上不要乘常数、不要加门控、不要放卷积或 dropout；相加之后也不要再接 ReLU 或 BN。这样，信号在前向和反向传播时，都能从任何一个单元直接到达另一个单元。\n\n在实现上，这篇的改动很小：把残差分支里 BN、ReLU 的位置往前挪，相加之后什么都不做。要训练很深的 ResNet，或者模型已经开始过拟合，可以优先试试 pre-activation 版本；网络浅一些的话，两种差别不大。\n\n「捷径上什么都别放」这个原则，几年后在 Transformer 的 LayerNorm 上又被讨论了一次，有兴趣可以接着看 [Pre-LN](/blog/papers/transformers/pre-ln) 那篇。\n","imageSizes":{"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png":{"width":967,"height":469},"/content/papers/assets/7710a267ec379977c4b41b07fbb2532dd44111584b6c0542b7ce9e96556e4f13.png":{"width":690,"height":659},"/content/papers/assets/f032506b6dda998779a8abf2fee9cbf44371ac4a2c4ebf8c93d834286bbb7501.png":{"width":965,"height":466},"/content/papers/assets/1a7a520a803ed65de475f2ee4bc5ade9b1833de9aefd758c5ef57153e5d087d2.png":{"width":967,"height":546}}}