{"notes":[{"id":"classic-cnns/stochastic-depth","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Stochastic Depth","subtitle":"今天这层请假","author":"Z. Yuan","paperDate":"2016-03-30","publishedOn":"2026-03-04","world":"external/stochastic-depth","paperTitle":"Deep Networks with Stochastic Depth","paperUrl":"https://arxiv.org/abs/1603.09382","bibliography":{"title":"Deep Networks with Stochastic Depth","authors":["Gao Huang","Yu Sun","Zhuang Liu","Daniel Sedra","Kilian Weinberger"],"url":"https://arxiv.org/abs/1603.09382","pdfUrl":"https://arxiv.org/pdf/1603.09382v3","codeUrl":null,"preprint":{"id":"1603.09382","submittedOn":"2016-03-30","url":"https://arxiv.org/abs/1603.09382"},"publication":null,"citation":"@misc{classiccnnsstochasticdepth2016,\n  title = {{Deep Networks with Stochastic Depth}},\n  author = {Gao Huang and Yu Sun and Zhuang Liu and Daniel Sedra and Kilian Weinberger},\n  year = {2016},\n  eprint = {1603.09382},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.09382}\n}\n","citationPath":"/content/papers/citations/classic-cnns/stochastic-depth.3716109ef477.bib"},"thumbnail":{"src":"/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png","width":963,"height":339},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/stochastic-depth.0e4d544484bf.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"让残差块随机缺席","text":"让残差块随机缺席","depth":3},{"id":"前面的层更常出席","text":"前面的层更常出席","depth":3},{"id":"测试时要乘上存活概率","text":"测试时要乘上存活概率","depth":3},{"id":"为什么有效","text":"为什么有效？","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"错误率和训练时间","text":"错误率和训练时间","depth":3},{"id":"一千两百层","text":"一千两百层","depth":3},{"id":"imagenet-上没有赢","text":"ImageNet 上没有赢","depth":3},{"id":"梯度真的更强吗","text":"梯度真的更强吗？","depth":3},{"id":"存活概率要设多少","text":"存活概率要设多少？","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""},{"id":"classic-cnns/densenet","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"卷积网络 100 层","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnet.98d0474f71e6.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"什么是残差","text":"什么是残差？","depth":3},{"id":"残差学习","text":"残差学习","depth":3},{"id":"残差模块","text":"残差模块","depth":3},{"id":"网络架构","text":"网络架构","depth":3},{"id":"参数配置","text":"参数配置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"更深的瓶颈架构","text":"更深的瓶颈架构","depth":3},{"id":"在-imagenet-上的表现","text":"在 ImageNet 上的表现","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"如何训练更深的卷积网络"},{"id":"classic-cnns/densenet","relation":"next","topic":"如何训练更深的卷积网络"}]},{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"捷径要干净","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnet-v2.9e686e0339ca.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"两个恒等映射","text":"两个恒等映射","depth":3},{"id":"捷径上多一点东西会怎样","text":"捷径上多一点东西会怎样？","depth":3},{"id":"相加之后的-relu-该放在哪","text":"相加之后的 ReLU 该放在哪？","depth":3},{"id":"换个角度看pre-activation","text":"换个角度看：pre-activation","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"越深差距越大","text":"越深，差距越大","depth":3},{"id":"和其他方法比较","text":"和其他方法比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]},{"id":"classic-cnns/densenet","category":"classic-cnns","legacyPath":"classic-cnns/1608-densenet","legacySlug":"1608-densenet","legacyPaths":[],"title":"DenseNet","subtitle":"全部连起来","author":"Z. Yuan","paperDate":"2016-08-25","publishedOn":"2024-07-02","paperTitle":"Densely Connected Convolutional Networks","paperUrl":"https://arxiv.org/abs/1608.06993","bibliography":{"title":"Densely Connected Convolutional Networks","authors":["Gao Huang","Zhuang Liu","Laurens van der Maaten","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1608.06993","pdfUrl":"https://arxiv.org/pdf/1608.06993","codeUrl":null,"preprint":{"id":"1608.06993","submittedOn":"2016-08-25","url":"https://arxiv.org/abs/1608.06993"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1608.06993"},"citation":"@misc{classiccnnsdensenet2016,\n  title = {{Densely Connected Convolutional Networks}},\n  author = {Gao Huang and Zhuang Liu and Laurens van der Maaten and Kilian Q. Weinberger},\n  year = {2016},\n  eprint = {1608.06993},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1608.06993}\n}\n","citationPath":"/content/papers/citations/classic-cnns/densenet.6f28153b67f7.bib"},"thumbnail":{"src":"/content/papers/assets/4f073a0fd1fc5b7415ca19c48934d180f9043fddc08301819a50ceb1b0ebfc69.jpg","width":1184,"height":828},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/densenet.c4f2eed4ee7c.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"网络架构","text":"网络架构","depth":3},{"id":"平均池化","text":"平均池化","depth":3},{"id":"growth-rate","text":"Growth Rate","depth":3},{"id":"bottleneck-layers","text":"Bottleneck Layers","depth":3},{"id":"compression","text":"Compression","depth":3},{"id":"实作细节","text":"实作细节","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验结果","text":"实验结果","depth":3},{"id":"紧凑模型","text":"紧凑模型","depth":3},{"id":"特征重用","text":"特征重用","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"background","topic":"如何训练更深的卷积网络"}]}],"text":"训练很深的网络，有一个两难。\n\n浅的网络，信号前向、反向都走得顺，训练也快，但表达能力有限。深的网络表达能力强，可是梯度会消失，前面算出的特征会被一层层的权重冲刷掉，训练时间也跟着深度一起变长：152 层的 ResNet，用好几张 GPU 在 ImageNet 上也要训练好几周。\n\n如果训练时用浅的网络，测试时用深的网络呢？\n\n这听起来自相矛盾。2016 年 3 月，康奈尔大学的 Gao Huang、Yu Sun、Daniel Sedra、Kilian Q. Weinberger 与清华大学的 Zhuang Liu 发布了这篇论文，后来发表在 ECCV 2016。他们的做法是：网络照样建得很深，但训练时每个小批量都随机去掉一部分残差块，只留下捷径。\n\n## 定义问题\n\n作者列出了很深的网络会遇到的三个问题：\n\n1. **梯度消失**：梯度往回传的时候，一再乘上小的权重，到了前面几层就变得很小。\n2. **前向的特征逐渐流失**：输入的特征、或者前面层算出的特征，一再乘上随机初始化的权重，到后面就被「冲刷掉」了，后面的层很难找到有意义的梯度方向。\n3. **训练时间太长**：前向和反向传播的时间都和深度成正比。\n\nResNet 和 Highway Networks 用捷径处理了前两个问题，让特征和梯度可以绕过权重层。第三个问题还在：网络有多深，每一步就要算多少层。\n\n有没有办法让网络在训练时「平均起来」浅一点，测试时又保有全部的深度？\n\n\n> [!TIP]\n> \n> ResNet 的残差块，可以先看我们的笔记：\n> \n> - **[[15.12] ResNet：卷积网络 100 层](/blog/papers/classic-cnns/resnet)**\n\n\n## 解决问题\n\n### 让残差块随机缺席\n\n作者用的就是 ResNet 原本的残差块。第 $\\ell$ 个块的输出是：\n\n$$\nH_\\ell = \\mathrm{ReLU}\\big(f_\\ell(H_{\\ell-1}) + \\mathrm{id}(H_{\\ell-1})\\big)\n$$\n\n$f_\\ell$ 是「卷积、BN、ReLU、卷积、BN」，$\\mathrm{id}$ 是恒等捷径。\n\n随机深度只多了一个伯努利随机变量 $b_\\ell \\in \\{0, 1\\}$：\n\n$$\nH_\\ell = \\mathrm{ReLU}\\big(b_\\ell\\, f_\\ell(H_{\\ell-1}) + \\mathrm{id}(H_{\\ell-1})\\big)\n$$\n\n$b_\\ell = 1$ 时，这就是原本的残差块。$b_\\ell = 0$ 时，残差分支整个消失，只剩 $\\mathrm{ReLU}(H_{\\ell-1})$。\n\n这里有个小细节：$H_{\\ell-1}$ 是上一个块最后的 ReLU 输出，本来就不会是负的（第一个块的输入，也是一组「卷积、BN、ReLU」的输出），所以 $\\mathrm{ReLU}(H_{\\ell-1}) = H_{\\ell-1}$。缺席的块因此恰好是恒等映射，信号原封不动地传到下一个块。\n\n$b_\\ell = 1$ 的概率记为 $p_\\ell$，作者称之为**存活概率**。每个小批量重新抽一次，被抽中缺席的块，这一步完全不用算：不做前向、不做反向，也不更新权重。\n\n### 前面的层更常出席\n\n每个块的存活概率要怎么设？最简单的是全部一样。作者建议的是线性递减：输入端是 $p_0 = 1$，最后一个块是 $p_L$，中间按比例递减：\n\n$$\np_\\ell = 1 - \\frac{\\ell}{L}(1 - p_L)\n$$\n\n[![上半部分是一排长条，标示 1.0、0.9、0.8、0.7、0.6、0.5，蓝色部分代表存活概率，从左到右逐渐变短，旁边的小图示说明了 active 与 inactive 两种块状态。下半部分是一张花的照片作为 Input，接着是五个残差块 f1 到 f5 与中间的 H1 到 H4；f3 和 f5 被红色叉号划掉，表示这一次被跳过](/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png)](/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png)\n\n*原论文 Fig. 2，取自 [**v3 PDF 第 5 页**](https://arxiv.org/pdf/1603.09382v3#page=5)。$p_0 = 1$、$p_L = 0.5$ 时的线性递减；第一个块的输入视为永远存在的 $H_0$。*\n\n作者的理由是：前面的层提取低级特征，后面的层都要用到，应该更稳定地出席。后面的实验也显示线性递减比固定概率好；而且结果对 $p_L$ 不太敏感，所以他们全文都用 $p_L = 0.5$。\n\n以 CIFAR 实验常用的 110 层 ResNet 为例，它有 $L = 54$ 个残差块：\n\n```python\nimport torch\n\nL, p_L = 54, 0.5                                  # ResNet-110：54 个残差块\np = torch.tensor([1 - l / L * (1 - p_L) for l in range(1, L + 1)])\nprint(\"前三个块的存活概率：\", [f\"{v:.3f}\" for v in p[:3]], \"；最后三个：\", [f\"{v:.3f}\" for v in p[-3:]])\nprint(\"期望深度：\", round(p.sum().item(), 2), \"；公式 (3L-1)/4 =\", (3 * L - 1) / 4)\n\ntorch.manual_seed(0)\nactive = torch.bernoulli(p.expand(10000, L)).sum(dim=1)   # 抽取 10000 个小批量\nprint(\"实际抽样：平均\", round(active.mean().item(), 2), \"个块，最少\", int(active.min()), \"，最多\", int(active.max()))\nprint(\"可能的子网络数：2^54 =\", f\"{2 ** L:.3e}\")\n```\n\n```text\n前三个块的存活概率： ['0.991', '0.981', '0.972'] ；最后三个： ['0.519', '0.509', '0.500']\n期望深度： 40.25 ；公式 (3L-1)/4 = 40.25\n实际抽样：平均 40.25 个块，最少 28 ，最多 51\n可能的子网络数：2^54 = 1.801e+16\n```\n\n训练时，平均每次只用到约 40 个块，测试时是完整的 54 个。期望深度是 $\\sum_\\ell p_\\ell = (3L-1)/4$，大约是 $3L/4$；少算的四分之一，就是训练时间省下的部分。每次抽到的网络深度也相差很大，从 28 到 51 个块都有，有些更新会走过特别短的网络，梯度到达前面几层的路径也更直接。\n\n### 测试时要乘上存活概率\n\n测试时，所有块都要出席，才能用上整个网络的容量。可是训练时，$f_\\ell$ 只在比例为 $p_\\ell$ 的更新里出现，下一层的权重是按这个比例校准的。所以测试时要把每个块的输出乘上它的存活概率：\n\n$$\nH_\\ell^{\\text{Test}} = \\mathrm{ReLU}\\big(p_\\ell\\, f_\\ell(H_{\\ell-1}^{\\text{Test}}; W_\\ell) + H_{\\ell-1}^{\\text{Test}}\\big)\n$$\n\n我们写一个块，比较训练和测试时的输出。为了看得清楚，这里的残差函数用一个线性层，也先不加最后的 ReLU：\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass StochasticDepthBlock(nn.Module):\n    def __init__(self, f, p):\n        super().__init__()\n        self.f, self.p = f, p                      # p：这个块的存活概率\n\n    def forward(self, x):\n        if self.training:\n            if torch.rand(()) < self.p:            # 存活：照常计算残差\n                return x + self.f(x)\n            return x                               # 没存活：整个块变成恒等映射\n        return x + self.p * self.f(x)              # 测试：乘上存活概率\n\ntorch.manual_seed(0)\nblock = StochasticDepthBlock(nn.Linear(4, 4), p=0.6)\nx = torch.randn(1, 4)\n\nblock.train()\nwith torch.no_grad():\n    outs = torch.stack([block(x) for _ in range(20000)])\nblock.eval()\nwith torch.no_grad():\n    test = block(x)\n\nprint(\"训练时的输出种类：\", torch.unique(outs.round(decimals=4), dim=0).shape[0], \"种\")\nprint(\"训练输出的平均：\", [f\"{v:.3f}\" for v in outs.mean(0).squeeze()])\nprint(\"测试时的输出：  \", [f\"{v:.3f}\" for v in test.squeeze()])\n```\n\n```text\n训练时的输出种类： 2 种\n训练输出的平均： ['-0.479', '0.711', '-1.017', '1.223']\n测试时的输出：   ['-0.479', '0.713', '-1.017', '1.223']\n```\n\n训练时，这个块的输出只有两种：有残差、没有残差。两万次的平均，和测试时乘上 $p = 0.6$ 的输出几乎一样。\n\n\n> [!NOTE]\n> \n> 现在的库大多把校正挪到训练时做，和 inverted dropout 一样：存活时把残差除以存活概率，测试时就什么都不用改。例如 torchvision 的 `torchvision.ops.stochastic_depth`，参数 `p` 是「被丢弃」的概率，存活时把输入除以 $1-p$。两种做法的期望值相同，读代码时要留意 `p` 指的是哪一个。\n\n\n### 为什么有效？\n\n作者给出了三个理由。\n\n- **训练变快**：被跳过的块不用算，前向和反向传播的时间跟着期望深度走，而不是完整深度。\n- **梯度更强**：训练时的网络变短，梯度要经过的层数变少，前面几层收到的梯度更大。\n- **隐式集成**：54 个块，每个出席或缺席，一共有 $2^{54}$ 种组合。每个小批量抽一个共享权重的子网络来更新，测试时再用上面的式子把它们合成一个网络。ResNet 论文里破纪录的成绩，正是由不同深度的 ResNet 集成得到的。\n\n这种集成的说法和 dropout 很像。区别在于 dropout 让网络变**窄**，随机深度让网络变**浅**。作者也提到，dropout 搭配 batch normalization 时效果常常会打折扣：他们在加了 BN 的 110 层 ResNet 上试了各种 dropout 比例，几乎没有改善；随机深度在同样的网络上仍然有效。\n\n## 讨论\n\n### 错误率和训练时间\n\n| 方法 | CIFAR-10+ | CIFAR-100+ | SVHN | ImageNet |\n| --- | ---: | ---: | ---: | ---: |\n| Highway Network | 7.60 | 32.24 | – | – |\n| Gen. Pool | 6.05 | 28.02 | 1.69 | – |\n| ResNet，固定深度 | 6.41 | 27.76 | 1.80 | 21.78 |\n| ResNet，随机深度 | **5.25** | **24.98** | 1.75 | 21.98 |\n\n*表：摘自原论文 Table 1 的部分行，测试错误率（%）；「+」表示使用标准的数据增强。「固定深度」是作者自己复现的 He 等人的实验。*\n\nCIFAR-10 用的是 110 层 ResNet，训练 500 个 epoch。固定深度的错误率是 6.41%，加上随机深度降到 5.25%，相对改善 18%；训练和网络设置完全相同，只差在训练时随机跳过残差块。CIFAR-100 用同一个网络，从 27.76% 降到 24.98%。\n\n[![两张训练曲线。左边是 CIFAR-10 上的 110 层 ResNet，右边是 CIFAR-100。红线是固定深度的测试错误率，蓝线是随机深度的测试错误率，另外两条细线是两者的训练损失。在第 250 个 epoch 学习率下降后，蓝线明显低于红线；左图标出 6.41% 与 5.25%，右图标出 27.88% 与 24.98%](/content/papers/assets/757bc4adb848efac2620e3a37c465e58ddc4a4c93e54f3b664bcea91cab64ffe.png)](/content/papers/assets/757bc4adb848efac2620e3a37c465e58ddc4a4c93e54f3b664bcea91cab64ffe.png)\n\n*原论文 Fig. 3，取自 [**v3 PDF 第 8 页**](https://arxiv.org/pdf/1603.09382v3#page=8)。CIFAR-10（左）与 CIFAR-100（右）的测试错误率与训练损失，圆点是验证错误率最低的那个 epoch。*\n\n学习率第一次下降之后，固定深度的测试错误率基本就停住了，随机深度还在往下走。作者也注意到，随机深度的测试错误率波动更大，推测是因为训练时深度一直在变。\n\n不用数据增强时，随机深度在 CIFAR-10 和 CIFAR-100 上也都有大约 15% 的相对改善：13.63% 降到 11.66%，44.74% 降到 37.8%。SVHN 用 152 层的网络，固定深度到了第二阶段就开始过拟合，错误率 1.80%；随机深度是 1.75%。\n\n训练时间的比较：\n\n| | CIFAR-10+ | CIFAR-100+ | SVHN |\n| --- | ---: | ---: | ---: |\n| 固定深度 | 20 小时 42 分 | 20 小时 51 分 | 33 小时 43 分 |\n| 随机深度 | 15 小时 7 分 | 15 小时 20 分 | 25 小时 33 分 |\n\n*表：摘自原论文 Table 2，$p_L = 0.5$ 的线性递减。*\n\n每一组都省下了大约四分之一的时间，和前面算出的期望深度一致。\n\n### 一千两百层\n\nResNet 论文试过 1202 层的网络，结果过拟合，测试错误率 7.93%，比 110 层还差。作者用同样的 1202 层网络重做了这个实验，训练 300 个 epoch，前 10 个 epoch 用 0.01 的学习率预热。\n\n固定深度的 1202 层网络错误率 6.67%，同样比 110 层的 6.41% 差。换成随机深度，错误率是 4.91%：比固定深度的 1202 层相对改善 27%，也比随机深度的 110 层（5.25%）更好，而且没有过拟合的迹象。作者说，这是第一次有超过一千层的网络，能比浅一些的网络进一步降低测试错误率。\n\n\n> [!TIP]\n> \n> 同一个月，ResNet 的作者走了另一条路，把残差单元改成 pre-activation，1001 层的 ResNet 在 CIFAR-10 上做到 4.92%（批量大小 64 时为 4.62%）：\n> \n> - **[[16.03] ResNet v2：捷径要干净](/blog/papers/classic-cnns/resnet-v2)**\n\n\n### ImageNet 上没有赢\n\nImageNet 用的是 152 层的 ResNet。固定深度训练 90 个 epoch，错误率 23.06%；随机深度在第 90 个 epoch 是 23.38%，略差一点。作者观察到随机深度的验证错误率还在明显下降，而且它省下了 25% 的计算，多训练 30 个 epoch，总时间也和固定深度的 90 个 epoch 差不多。多训练之后，随机深度到了 21.98%；可是固定深度也多跑了 30 个 epoch，到了 21.78%。\n\n所以在这个设置下，随机深度没有带来准确率上的好处。作者引用了一位匿名审稿人的说法：ImageNet 的模型和 CIFAR 的模型还处在不同的状态；ImageNet 需要的模型复杂度可能远超 152 层的 ResNet，随机深度也许要等到更大的模型才会显出效果。\n\n### 梯度真的更强吗？\n\n[![折线图，横轴是 epoch（0 到 500），纵轴是第一个卷积层的平均梯度大小（单位为 10 的负 5 次方）。红线是固定深度，蓝线是随机深度。两条虚线标出第 250 与 375 个 epoch 的学习率下降，梯度在这两处都大幅下降；蓝线始终高于红线，学习率下降之后差距更明显](/content/papers/assets/6e62eb722549bcdd22e104076055f0d6e70ad5e66c660f0e827fcfa67318c9a1.png)](/content/papers/assets/6e62eb722549bcdd22e104076055f0d6e70ad5e66c660f0e827fcfa67318c9a1.png)\n\n*原论文 Fig. 7，取自 [**v3 PDF 第 12 页**](https://arxiv.org/pdf/1603.09382v3#page=12)。CIFAR-10 上，第一个残差块的第一个卷积层每个 epoch 的平均梯度大小；虚线是学习率除以 10 的时间点。*\n\n两次大幅下降来自学习率的调度。随机深度的梯度一直更大，学习率下降之后差距更明显，这支持了作者「缓解梯度消失」的说法。\n\n### 存活概率要设多少？\n\n[![左图：CIFAR-10 上 110 层 ResNet 的测试错误率随存活概率 p_L 的变化。实线是线性递减，虚线是固定概率，红色虚线是固定深度的 6.41%。线性递减的曲线在 p_L 为 0.5 附近最低，约 5.25%，标注 1.33x 的加速；p_L 为 0.2 时仍接近固定深度，标注 1.67x；固定概率的曲线大多较高，只在 p_L 约 0.8 处低于线性递减，在 p_L 为 0.5 处标注 2x。右图：热力图，纵轴是 p_L（1.0 到 0.5），横轴是网络深度（20 到 110 层），格子里是测试错误率；20 层时 p_L 在 0.9 到 0.7 之间比固定深度好，再小就变差，110 层时 p_L 为 0.5 最好，最低的值大致沿着对角线分布](/content/papers/assets/1f923643ee1fd74e75ef508bccb935e721a1b51f610ca3dcbac82168dc0573ad.png)](/content/papers/assets/1f923643ee1fd74e75ef508bccb935e721a1b51f610ca3dcbac82168dc0573ad.png)\n\n*原论文 Fig. 8，取自 [**v3 PDF 第 13 页**](https://arxiv.org/pdf/1603.09382v3#page=13)。左：110 层 ResNet 在不同 $p_L$ 下的 CIFAR-10 测试错误率；右：不同 $p_L$ 与网络深度下的测试错误率，$p_L = 1$ 那一行就是固定深度。*\n\n$p_L$ 是这个方法唯一的超参数。从左图可以看到：\n\n1. 只要 $p_L$ 设得合理，两种规则都比固定深度好。\n2. 在大多数的 $p_L$ 下，线性递减都比固定概率好。\n3. 线性递减对 $p_L$ 不太敏感，$p_L$ 在 0.4 到 0.8 之间都有不错的结果。\n4. 即使 $p_L = 0.2$，线性递减的错误率也和固定深度相当，训练时间却少了 40%。\n\n右边的热力图加入了深度。20 层的网络，只跳过一点（$p_L$ 在 0.9 到 0.7）时比固定深度好，跳过得再多就变差：$p_L = 0.5$ 时错误率从 8.31% 升到 9.39%。110 层的网络则是 $p_L = 0.5$ 最好。表现最好的格子大致沿着对角线分布：网络越深，越能承受更激进的跳过。网络要足够深，随机深度才会明显胜过固定深度；浅一些的网络，跳过得保守一点也还有帮助。\n\n## 结论\n\n随机深度是一种训练方法，不改变测试时的网络：训练时随机跳过残差块，期望深度变成约四分之三，测试时再用存活概率把所有块合起来。实验上，CIFAR 与 SVHN 的错误率都下降了，训练时间少了约四分之一，1202 层的网络也不再过拟合。ImageNet 上的 152 层 ResNet 则没有看到准确率上的好处。\n\n在一些代码库里，它以 DropPath 的名字出现。本文的 Gao Huang、Zhuang Liu 与 Kilian Q. Weinberger 几个月后提出了 DenseNet，论文开头就引用了随机深度：既然随机跳过层也能训练，残差网络里很多层的贡献其实很小。\n\n\n> [!TIP]\n> \n> DenseNet 可以看我们的笔记：\n> \n> - **[[16.08] DenseNet：全部连起来](/blog/papers/classic-cnns/densenet)**\n\n\n随机深度用起来很简单：训练时，每个残差分支乘上一个随机的 0 或 1；测试时，换成存活概率。现在的库大多改成训练时除以存活概率，像 torchvision 的 `stochastic_depth`，测试时就什么都不用改。\n\n作者当年猜测，也许要等到更大的模型，随机深度才会在 ImageNet 上显出好处。后来 [ConvNeXt](/blog/papers/classic-cnns/convnext)、[DINOv2](/blog/papers/contrastive-learning/dinov2) 这些大模型的训练配方里，确实都用上了它。\n","imageSizes":{"/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png":{"width":963,"height":339},"/content/papers/assets/757bc4adb848efac2620e3a37c465e58ddc4a4c93e54f3b664bcea91cab64ffe.png":{"width":965,"height":372},"/content/papers/assets/6e62eb722549bcdd22e104076055f0d6e70ad5e66c660f0e827fcfa67318c9a1.png":{"width":756,"height":345},"/content/papers/assets/1f923643ee1fd74e75ef508bccb935e721a1b51f610ca3dcbac82168dc0573ad.png":{"width":973,"height":422}}}