{"notes":[{"id":"classic-cnns/inception-v4","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v4","subtitle":"快很多，准一点","author":"Z. Yuan","paperDate":"2016-02-23","publishedOn":"2026-01-27","world":"external/inception-v4","paperTitle":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","paperUrl":"https://arxiv.org/abs/1602.07261","bibliography":{"title":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","authors":["Christian Szegedy","Sergey Ioffe","Vincent Vanhoucke","Alex Alemi"],"url":"https://arxiv.org/abs/1602.07261","pdfUrl":"https://arxiv.org/pdf/1602.07261v2","codeUrl":null,"preprint":{"id":"1602.07261","submittedOn":"2016-02-23","url":"https://arxiv.org/abs/1602.07261"},"publication":null,"citation":"@misc{classiccnnsinceptionv42016,\n  title = {{Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning}},\n  author = {Christian Szegedy and Sergey Ioffe and Vincent Vanhoucke and Alex Alemi},\n  year = {2016},\n  eprint = {1602.07261},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1602.07261}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v4.ed8e0294ae0b.bib"},"thumbnail":{"src":"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png","width":666,"height":656},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inception-v4.615919e732a6.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"卸下包袱inception-v4","text":"卸下包袱：Inception-v4","depth":3},{"id":"进场的那一段","text":"进场的那一段","depth":3},{"id":"把残差接进-inception","text":"把残差接进 Inception","depth":3},{"id":"缩小残差稳住训练","text":"缩小残差，稳住训练","depth":3},{"id":"训练设置","text":"训练设置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"残差让训练快了很多","text":"残差让训练快了很多","depth":3},{"id":"单个模型的成绩","text":"单个模型的成绩","depth":3},{"id":"集成","text":"集成","depth":3},{"id":"残差连接是必需的吗","text":"残差连接是「必需」的吗？","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"卷积网络 100 层","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnet.98d0474f71e6.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"什么是残差","text":"什么是残差？","depth":3},{"id":"残差学习","text":"残差学习","depth":3},{"id":"残差模块","text":"残差模块","depth":3},{"id":"网络架构","text":"网络架构","depth":3},{"id":"参数配置","text":"参数配置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"更深的瓶颈架构","text":"更深的瓶颈架构","depth":3},{"id":"在-imagenet-上的表现","text":"在 ImageNet 上的表现","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"如何训练更深的卷积网络"},{"id":"classic-cnns/densenet","relation":"next","topic":"如何训练更深的卷积网络"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"大卷积核退场","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2025-12-16","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":null,"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inception-v3.847c152f5969.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"把-55-拆成两层-33","text":"把 5×5 拆成两层 3×3","depth":3},{"id":"再拆成-1n-与-n1","text":"再拆成 1×n 与 n×1","depth":3},{"id":"辅助分类器的真正作用","text":"辅助分类器的真正作用","depth":3},{"id":"缩小特征图又不想卡住信息","text":"缩小特征图，又不想卡住信息","depth":3},{"id":"把零件组装起来","text":"把零件组装起来","depth":3},{"id":"让模型别太有把握","text":"让模型别太有把握","depth":3},{"id":"训练设置","text":"训练设置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"一项一项加上去","text":"一项一项加上去","depth":3},{"id":"分辨率低一点差多少","text":"分辨率低一点，差多少？","depth":3},{"id":"和其他模型比较","text":"和其他模型比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"捷径要干净","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnet-v2.9e686e0339ca.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"两个恒等映射","text":"两个恒等映射","depth":3},{"id":"捷径上多一点东西会怎样","text":"捷径上多一点东西会怎样？","depth":3},{"id":"相加之后的-relu-该放在哪","text":"相加之后的 ReLU 该放在哪？","depth":3},{"id":"换个角度看pre-activation","text":"换个角度看：pre-activation","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"越深差距越大","text":"越深，差距越大","depth":3},{"id":"和其他方法比较","text":"和其他方法比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]}],"text":"ILSVRC 2015 的分类冠军是微软研究院的 ResNet。它的架构很传统，一层层卷积叠上去，只多了残差连接；单个模型的成绩，和 Google 刚发表的 Inception-v3 差不多。\n\n两条路线各有一套做法：Inception 靠多分支的模块和精打细算的通道数，把计算量压低；ResNet 靠残差连接，让很深的网络也训练得动。\n\n那把两者合在一起呢？\n\n2016 年 2 月，Google 的 Christian Szegedy、Sergey Ioffe、Vincent Vanhoucke 与 Alex Alemi 发布了这篇论文，后来发表在 AAAI 2017。标题把三件事都写进去了：新的 Inception-v4、加上残差连接的 Inception-ResNet，以及残差连接对训练到底有什么影响。\n\n\n> [!TIP]\n> \n> 这篇的两个前作，可以先看我们的笔记：\n> \n> - **[[15.12] ResNet：卷积网络 100 层](/blog/papers/classic-cnns/resnet)**\n> - **[[15.12] Inception-v3：大卷积核退场](/blog/papers/classic-cnns/inception-v3)**\n\n\n## 定义问题\n\n这篇论文同时处理两个问题。\n\n第一个是残差连接。ResNet 的作者主张，残差连接是训练非常深的卷积网络**本质上必需**的东西。Inception 网络本来就很深，把 Inception 模块最后的「通道拼接」换成「加回输入」，看起来很自然。问题是：这样做有什么好处？好处体现在最终的准确率上，还是在别的地方？\n\n第二个是 Inception 自己。作者坦白地说，Inception-v3 背了很多历史包袱。早期的模型用 DistBelief 做分布式训练，每个副本要切成好几个子网络才放得进内存；为了让各个子网络的计算量平衡，各层的滤波器数量都经过仔细调整。换到 TensorFlow 之后，这个限制就没有了。另一方面，他们过去改架构一向保守，一次只动一个组件，其余不变，结果网络越长越复杂。\n\n既然包袱已经可以放下，Inception 能不能变得更简单、更深、更宽？\n\n## 解决问题\n\n### 卸下包袱：Inception-v4\n\n[![Inception-v4 的整体架构，自下而上：Input 299x299x3、Stem（输出 35x35x384）、4 x Inception-A（35x35x384）、Reduction-A（17x17x1024）、7 x Inception-B（17x17x1024）、Reduction-B（8x8x1536）、3 x Inception-C（8x8x1536）、Average Pooling（1536）、Dropout（keep 0.8）、Softmax（1000）](/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png)](/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png)\n\n*原论文 Figure 9，取自 [**v2 PDF 第 5 页**](https://arxiv.org/pdf/1602.07261v2#page=5)。Inception-v4 的整体架构，右侧是每一段的输出大小。*\n\n这张图和 Inception-v3 的架构表很像：三种分辨率（$35\\times35$、$17\\times17$、$8\\times8$），每一段叠几个 Inception 模块，中间用缩小模块切换到下一个分辨率。区别在于 Inception-v4 **同一个分辨率只用一种模块**，数量也变多了：4 个 Inception-A、7 个 Inception-B、3 个 Inception-C。\n\n三种模块延续了 Inception-v3 的分解方法：\n\n- **Inception-A**（$35\\times35$）：四条分支，最长的一条用两层 $3\\times3$ 取代 $5\\times5$，每条输出 96 个通道，合起来 384。\n- **Inception-B**（$17\\times17$）：用 $1\\times7$ 和 $7\\times1$ 的非对称卷积，合起来 1024 个通道。\n- **Inception-C**（$8\\times8$）：把 $1\\times3$ 和 $3\\times1$ 并行输出，合起来 1536 个通道。\n\n图上标了「V」的卷积是 valid padding，不补零，输出会变小；没有标的都是 same padding，输出大小和输入一样。\n\n### 进场的那一段\n\n和 Inception-v3 相比，最明显的改变在网络的开头。作者把它叫做 stem：\n\n[![Inception-v4 的 stem，自下而上：Input 299x299x3；3x3 Conv（32 stride 2 V）得到 149x149x32；3x3 Conv（32 V）得到 147x147x32；3x3 Conv（64）得到 147x147x64；接着分成 3x3 MaxPool（stride 2 V）与 3x3 Conv（96 stride 2 V）两条，Filter concat 得到 73x73x160；再分成两条：1x1 Conv（64）接 3x3 Conv（96 V），以及 1x1 Conv（64）、7x1 Conv（64）、1x7 Conv（64）、3x3 Conv（96 V），Filter concat 得到 71x71x192；最后分成 3x3 Conv（192 V）与 MaxPool（stride=2 V）两条，Filter concat 得到 35x35x384](/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png)](/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png)\n\n*原论文 Figure 3，取自 [**v2 PDF 第 3 页**](https://arxiv.org/pdf/1602.07261v2#page=3)。Inception-v4 与 Inception-ResNet-v2 共用的 stem。*\n\n前三层和 Inception-v3 一样，从 $299\\times299$ 走到 $147\\times147\\times64$。之后连续三次「分开走、再拼起来」，每一次都有一条分支负责缩小，或者两条分支各自处理再合并。我们照着图走一遍：\n\n```python\nimport torch\nimport torch.nn as nn\n\ndef conv(cin, cout, k, stride=1, valid=False):\n    # 标了 V 的卷积不补零；其他卷积补零，让输出大小和输入相同\n    pad = 0 if valid else (k[0] // 2, k[1] // 2) if isinstance(k, tuple) else k // 2\n    return nn.Sequential(nn.Conv2d(cin, cout, k, stride, pad), nn.ReLU())\n\nx = torch.randn(1, 3, 299, 299)\n\nx = conv(3, 32, 3, stride=2, valid=True)(x)\nx = conv(32, 32, 3, valid=True)(x)\nx = conv(32, 64, 3)(x)\nprint(\"三层卷积后：\", tuple(x.shape[1:]))\n\n# 第一次并行：最大池化 | 步幅 2 的卷积\na = nn.MaxPool2d(3, 2)(x)\nb = conv(64, 96, 3, stride=2, valid=True)(x)\nx = torch.cat([a, b], dim=1)\nprint(\"池化\", tuple(a.shape[1:]), \"+ 卷积\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n\n# 第二次并行：两条卷积分支\na = nn.Sequential(conv(160, 64, 1), conv(64, 96, 3, valid=True))(x)\nb = nn.Sequential(conv(160, 64, 1), conv(64, 64, (7, 1)), conv(64, 64, (1, 7)), conv(64, 96, 3, valid=True))(x)\nx = torch.cat([a, b], dim=1)\nprint(\"分支一\", tuple(a.shape[1:]), \"+ 分支二\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n\n# 第三次并行：步幅 2 的卷积 | 最大池化\na = conv(192, 192, 3, stride=2, valid=True)(x)    # 要和池化的 35×35 拼接，步幅必须是 2\nb = nn.MaxPool2d(3, 2)(x)\nx = torch.cat([a, b], dim=1)\nprint(\"卷积\", tuple(a.shape[1:]), \"+ 池化\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n```\n\n```text\n三层卷积后： (64, 147, 147)\n池化 (64, 73, 73) + 卷积 (96, 73, 73) → (160, 73, 73)\n分支一 (96, 71, 71) + 分支二 (96, 71, 71) → (192, 71, 71)\n卷积 (192, 35, 35) + 池化 (192, 35, 35) → (384, 35, 35)\n```\n\n输出的形状和图上标的 $73\\times73\\times160$、$71\\times71\\times192$、$35\\times35\\times384$ 一致。池化不改变通道数，所以第一次合并是 $64+96=160$，最后一次是 $192+192=384$。\n\n第一次和第三次合并，就是 Inception-v3 缩小特征图的方法：步幅 2 的卷积和步幅 2 的池化并行，避免先池化造成的信息瓶颈。这次连网络的开头也这样做了。\n\n### 把残差接进 Inception\n\n残差版本的 Inception，作者用的是更便宜的模块。以 Inception-ResNet-v2 的 $35\\times35$ 模块为例：\n\n[![Inception-ResNet-A 模块：下方的 Relu activation 分成四条路。最左边一条直接连到上方的加号；另外三条分支分别是 1x1 Conv（32）、1x1 Conv（32）接 3x3 Conv（32）、1x1 Conv（32）接 3x3 Conv（48）再接 3x3 Conv（64）。三条分支的输出一起进入 1x1 Conv（384 Linear），再和最左边的直通路径相加，最后经过 Relu activation](/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png)](/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png)\n\n*原论文 Figure 16，取自 [**v2 PDF 第 8 页**](https://arxiv.org/pdf/1602.07261v2#page=8)。Inception-ResNet-v2 的 $35\\times35$ 模块（Inception-ResNet-A）。*\n\n和纯 Inception 模块相比，有两个地方不一样：\n\n1. 分支比较窄。三条分支最后只有 $32+32+64=128$ 个通道，远少于输入的 384。\n2. 分支拼接之后，多了一层**没有激活函数**的 $1\\times1$ 卷积，把通道数扩回 384，才能和输入相加。作者把它叫做 filter-expansion layer，用来弥补 Inception 模块造成的降维。\n\n```python\nimport torch\nimport torch.nn as nn\n\ndef conv(cin, cout, k):\n    return nn.Sequential(nn.Conv2d(cin, cout, k, padding=k // 2, bias=False), nn.BatchNorm2d(cout), nn.ReLU())\n\nclass InceptionResNetA(nn.Module):\n    def __init__(self, channels=384, scale=0.1):\n        super().__init__()\n        self.b1 = conv(channels, 32, 1)\n        self.b2 = nn.Sequential(conv(channels, 32, 1), conv(32, 32, 3))\n        self.b3 = nn.Sequential(conv(channels, 32, 1), conv(32, 48, 3), conv(48, 64, 3))\n        self.expand = nn.Conv2d(32 + 32 + 64, channels, 1)   # 1×1，没有激活函数\n        self.scale = scale\n\n    def forward(self, x):\n        branches = torch.cat([self.b1(x), self.b2(x), self.b3(x)], dim=1)\n        residual = self.expand(branches)\n        print(\"分支拼接后：\", tuple(branches.shape[1:]), \"→ 1×1 扩回：\", tuple(residual.shape[1:]))\n        return torch.relu(x + self.scale * residual)\n\nx = torch.randn(2, 384, 35, 35)\nprint(\"输出：\", tuple(InceptionResNetA()(x).shape[1:]))\n```\n\n```text\n分支拼接后： (128, 35, 35) → 1×1 扩回： (384, 35, 35)\n输出： (384, 35, 35)\n```\n\n代码里的 `scale` 是下一节的主角，先放着。\n\n作者试了好几种残差版本，论文只详细列出了两个：\n\n| 网络 | 计算量对照 | Stem | 主体 |\n| --- | --- | --- | --- |\n| Inception-ResNet-v1 | 约等于 Inception-v3 | 自己的 stem（Figure 14） | 5 × A、Reduction-A、10 × B、Reduction-B、5 × C |\n| Inception-ResNet-v2 | 约等于 Inception-v4 | 和 Inception-v4 相同（Figure 3） | 同上，模块更宽 |\n\n*表：整理自原论文第 3.2 节与 Figure 15。两个网络的整体结构相同，模块的通道数不同。*\n\n虽然 Inception-ResNet-v2 和 Inception-v4 的原始计算量相当，作者也提到，Inception-v4 实际上每一步训练都慢很多，可能是因为层数更多。\n\n还有一个细节：Inception-ResNet 的 batch normalization 只加在普通的层上，**相加之后不加**。作者说，全面使用 batch normalization 应该会更好，但在激活值很大的那些层上，batch normalization 会占用不成比例的 GPU 显存；为了让每个模型副本放得进单张 GPU，他们拿掉了这些 BN，换来可以多叠很多模块。\n\n### 缩小残差，稳住训练\n\n接下来，作者遇到了一个麻烦：滤波器数量超过 1000 时，残差版本变得不稳定，网络会在训练初期「死掉」。具体症状是，训练几万次迭代之后，平均池化前的最后一层只输出 0。调低学习率、在这一层多加一个 batch normalization，都救不回来。\n\n有效的做法很简单：在相加之前，先把残差乘上一个小的常数。\n\n[![残差缩放的示意图：下方的 Relu activation 分成两条路，一条直接连到上方的加号，另一条经过 Inception 模块，再经过 Activation Scaling 模块，才连到加号；相加之后是 Relu activation](/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png)](/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png)\n\n*原论文 Figure 20，取自 [**v2 PDF 第 9 页**](https://arxiv.org/pdf/1602.07261v2#page=9)。在 Inception-ResNet 模块里缩放残差，常数通常取 0.1 左右。*\n\n作者一般选 0.1 到 0.3 之间的值。ResNet 论文遇到很深的网络不稳定时，用的是先以很低的学习率「预热」、再提高学习率的两阶段训练；作者发现滤波器很多时，连 0.00001 的学习率都挡不住不稳定，之后的高学习率还可能把预热的效果毁掉。相比之下，缩放残差可靠得多。即使原本不缩放也能训练，加上缩放也没有损害最终准确率，训练还更稳定。\n\n论文没有分析为什么会「死掉」。不过有一件事很容易看到：Inception-ResNet 的分支里有 batch normalization，输出的大小大致固定；主干的相加之后没有 batch normalization，每经过一个模块就累积一次。我们用一个简化的模型看看主干上的信号怎么变化：\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nx0 = torch.randn(4096, 256)                     # 主干上的信号：4096 个位置，256 个通道\n\nfor scale in (1.0, 0.3, 0.1):\n    x = x0.clone()\n    stds = []\n    for block in range(1, 41):\n        w = torch.randn(256, 256) / 16\n        r = torch.relu(x) @ w\n        r = (r - r.mean(0)) / r.std(0)          # 分支里有 BN：每个通道的输出被拉回标准差 1\n        x = x + scale * r                       # 主干的相加之后没有 BN\n        if block in (10, 20, 40):\n            stds.append(f\"{x.std().item():5.2f}\")\n    print(f\"缩放 {scale}：第 10、20、40 个模块后，主干的标准差\", \" \".join(stds))\n```\n\n```text\n缩放 1.0：第 10、20、40 个模块后，主干的标准差  3.31  4.57  6.42\n缩放 0.3：第 10、20、40 个模块后，主干的标准差  1.37  1.67  2.15\n缩放 0.1：第 10、20、40 个模块后，主干的标准差  1.05  1.10  1.18\n```\n\n不缩放时，每个模块都加进一份和主干差不多大的信号，40 个模块之后，标准差大约是 $\\sqrt{41}\\approx6.4$。缩放 0.1 时，每个模块只做一点小修正，主干上的信号主要还是从直通路径传下来的。这只是初始化时的简化示意，不能解释训练中途的「死掉」；论文能确定的只有实验结果：缩放之后，宽的残差网络训练得很稳。\n\n### 训练设置\n\n和 Inception-v3 几乎一样：TensorFlow 分布式训练，这次用 20 个副本，各跑在一张 NVIDIA Kepler GPU 上。最好的模型用 RMSProp（decay 0.9、$\\epsilon=1.0$），学习率 0.045，每两个 epoch 乘以 0.94；评估时使用参数的滑动平均。\n\n## 讨论\n\n### 残差让训练快了很多\n\n[![折线图，横轴是 epoch（20 到 200），纵轴是 top-1 错误率（15% 到 29%）。绿色虚线是 inception-v3，红色实线是 inception-resnet-v1。红线下降得快很多，大约 80 个 epoch 就接近 22%，绿线要到 160 个 epoch 左右才追上；最后两条线都停在 21% 多一点](/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png)](/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png)\n\n*原论文 Figure 21，取自 [**v2 PDF 第 9 页**](https://arxiv.org/pdf/1602.07261v2#page=9)。计算量相近的 Inception-v3 与 Inception-ResNet-v1，训练过程中的 top-1 错误率（单次裁剪）。*\n\n红线是加了残差的 Inception-ResNet-v1，绿线是 Inception-v3。红线下降得快很多，可是两条线最后停在差不多的地方：top-1 是残差版本略差一点，top-5 则是残差版本略好一点。Inception-v4 与 Inception-ResNet-v2 这一对，也是残差版本训练得更快，最后的成绩略好。\n\n把四个模型放在一起比，作者的结论是：残差版本收敛得更快，但最终准确率主要取决于模型大小。\n\n### 单个模型的成绩\n\n| 网络 | Top-1 错误率 | Top-5 错误率 |\n| --- | ---: | ---: |\n| BN-Inception | 25.2% | 7.8% |\n| Inception-v3 | 21.2% | 5.6% |\n| Inception-ResNet-v1 | 21.3% | 5.5% |\n| Inception-v4 | 20.0% | 5.0% |\n| **Inception-ResNet-v2** | **19.9%** | **4.9%** |\n\n*表：摘自原论文 Table 2。单个模型、单次裁剪，验证集排除了黑名单图像。*\n\n把计算量相近的两两分成一组来看：Inception-v3 和 Inception-ResNet-v1 几乎打平，Inception-v4 和 Inception-ResNet-v2 也只差 0.1 个百分点。两组之间的差距，比组内有没有残差的差距大得多。\n\n多次裁剪的结果也是同样的走势：\n\n| 网络 | 裁剪数 | Top-1 错误率 | Top-5 错误率 |\n| --- | ---: | ---: | ---: |\n| ResNet-152 | 10 | 21.4% | 5.7% |\n| Inception-v3 | 12 | 19.8% | 4.6% |\n| Inception-ResNet-v1 | 12 | 19.8% | 4.6% |\n| Inception-v4 | 12 | 18.7% | 4.2% |\n| Inception-ResNet-v2 | 12 | 18.7% | 4.1% |\n| ResNet-152 | dense | 19.4% | 4.5% |\n| Inception-v3 | 144 | 18.9% | 4.3% |\n| Inception-ResNet-v1 | 144 | 18.8% | 4.3% |\n| Inception-v4 | 144 | 17.7% | 3.8% |\n| Inception-ResNet-v2 | 144 | 17.8% | 3.7% |\n\n*表：摘自原论文 Table 3 与 Table 4，完整的 50,000 张验证集。*\n\n### 集成\n\n| 网络 | 模型数 | Top-1 错误率 | Top-5 错误率 |\n| --- | ---: | ---: | ---: |\n| ResNet-152 | 6 | – | 3.6% |\n| Inception-v3 | 4 | 17.3% | 3.6% |\n| **Inception-v4 + 3 × Inception-ResNet-v2** | **4** | **16.5%** | **3.1%** |\n\n*表：摘自原论文 Table 5，完整的 50,000 张验证集。ResNet 为 dense 评估，Inception 系列为 144 次裁剪。*\n\n一个 Inception-v4 加三个 Inception-ResNet-v2，验证集 top-5 错误率 3.1%。这组集成也提交到了测试服务器，得到 3.08%。作者特别说明，过去一年他们只向测试服务器提交过两次结果，一次是 BN-Inception 论文，一次是 ILSVRC 2015 的 CLSLOC 比赛，所以测试集上的数字可以看作对泛化能力的真实估计。\n\n作者也注意到，单个模型的进步，到了集成就没有那么明显了。\n\n### 残差连接是「必需」的吗？\n\n回到开头的第一个问题。作者的回答很直接：至少在图像识别上，他们的结果不支持 ResNet 论文「残差连接本质上必需」的说法。Inception-v4 没有残差连接，照样训练得起来，成绩也和 Inception-ResNet-v2 相当。\n\n他们也留了余地：要看清残差连接的全部好处，可能需要更深的架构、更多的实验点。能确定的是，残差连接让训练快了很多，光这一点就值得使用。\n\n## 结论\n\n这篇论文交出了三个网络：纯 Inception 的 Inception-v4，以及计算量分别对齐 Inception-v3、Inception-v4 的 Inception-ResNet-v1、v2。\n\n比较的条件要先说清楚：Inception-ResNet 的模块是「挑得有点随意」的（作者的原话是 somewhat ad hoc），只要求计算量和对应的纯 Inception 模型相近；作者也试过更大、更宽的版本，在 ImageNet 上的表现很接近。在这个条件下，残差连接带来的主要是训练速度；真正拉开差距的是模型变大，Inception-v4 和 Inception-ResNet-v2 都明显好于 v3 那一级的模型。\n\n这篇的结论对拿模型来用的人很实际：在意训练时间，就加上残差连接；在意最终的准确率，模型大小比有没有残差更重要。网络很宽时，记得先把残差乘上 0.1 到 0.3 再加回去，不然可能训练到一半就「死掉」。\n\n至于残差是不是训练深网络的必需品，ResNet 和这篇各有说法。同一时期，ResNet 的作者自己也在研究残差该怎么接才对，也就是下一篇的 [ResNet v2](/blog/papers/classic-cnns/resnet-v2)。\n","imageSizes":{"/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png":{"width":335,"height":941},"/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png":{"width":431,"height":1189},"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png":{"width":666,"height":656},"/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png":{"width":277,"height":396},"/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png":{"width":565,"height":356}}}