{"notes":[{"id":"classic-cnns/stochastic-depth","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Stochastic Depth","subtitle":"This Layer Is Taking the Day Off","author":"Z. Yuan","paperDate":"2016-03-30","publishedOn":"2026-03-04","world":"external/stochastic-depth","paperTitle":"Deep Networks with Stochastic Depth","paperUrl":"https://arxiv.org/abs/1603.09382","bibliography":{"title":"Deep Networks with Stochastic Depth","authors":["Gao Huang","Yu Sun","Zhuang Liu","Daniel Sedra","Kilian Weinberger"],"url":"https://arxiv.org/abs/1603.09382","pdfUrl":"https://arxiv.org/pdf/1603.09382v3","codeUrl":null,"preprint":{"id":"1603.09382","submittedOn":"2016-03-30","url":"https://arxiv.org/abs/1603.09382"},"publication":null,"citation":"@misc{classiccnnsstochasticdepth2016,\n  title = {{Deep Networks with Stochastic Depth}},\n  author = {Gao Huang and Yu Sun and Zhuang Liu and Daniel Sedra and Kilian Weinberger},\n  year = {2016},\n  eprint = {1603.09382},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.09382}\n}\n","citationPath":"/content/papers/citations/classic-cnns/stochastic-depth.3716109ef477.bib"},"thumbnail":{"src":"/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png","width":963,"height":339},"locale":"en","path":"/content/papers/en/classic-cnns/stochastic-depth.24f281d0d16d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"letting-blocks-randomly-sit-out","text":"Letting Blocks Randomly Sit Out","depth":3},{"id":"early-layers-show-up-more-often","text":"Early Layers Show Up More Often","depth":3},{"id":"multiply-by-the-survival-probability-at-test-time","text":"Multiply by the Survival Probability at Test Time","depth":3},{"id":"why-does-it-work","text":"Why Does It Work?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"error-rates-and-training-time","text":"Error Rates and Training Time","depth":3},{"id":"twelve-hundred-layers","text":"Twelve Hundred Layers","depth":3},{"id":"no-win-on-imagenet","text":"No Win on ImageNet","depth":3},{"id":"are-the-gradients-really-stronger","text":"Are the Gradients Really Stronger?","depth":3},{"id":"what-should-the-survival-probability-be","text":"What Should the Survival Probability Be?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""},{"id":"classic-cnns/densenet","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"Convolutional with 100 Layers","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"en","path":"/content/papers/en/classic-cnns/resnet.b65375cb9846.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-is-residual","text":"What is Residual?","depth":3},{"id":"residual-learning","text":"Residual Learning","depth":3},{"id":"residual-module","text":"Residual Module","depth":3},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"architectures-and-computation","text":"Architectures and Computation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"deeper-bottleneck-architectures","text":"Deeper Bottleneck Architectures","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"Training deeper convolutional networks"},{"id":"classic-cnns/densenet","relation":"next","topic":"Training deeper convolutional networks"}]},{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"Keep the Shortcut Clean","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"en","path":"/content/papers/en/classic-cnns/resnet-v2.1b56c20d8b4c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"two-identity-mappings","text":"Two Identity Mappings","depth":3},{"id":"what-if-the-shortcut-carries-something-extra","text":"What If the Shortcut Carries Something Extra?","depth":3},{"id":"where-should-the-after-addition-relu-go","text":"Where Should the After-Addition ReLU Go?","depth":3},{"id":"a-different-angle-pre-activation","text":"A Different Angle: Pre-Activation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-deeper-the-wider-the-gap","text":"The Deeper, the Wider the Gap","depth":3},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]},{"id":"classic-cnns/densenet","category":"classic-cnns","legacyPath":"classic-cnns/1608-densenet","legacySlug":"1608-densenet","legacyPaths":[],"title":"DenseNet","subtitle":"Connecting Everything","author":"Z. Yuan","paperDate":"2016-08-25","publishedOn":"2024-07-02","paperTitle":"Densely Connected Convolutional Networks","paperUrl":"https://arxiv.org/abs/1608.06993","bibliography":{"title":"Densely Connected Convolutional Networks","authors":["Gao Huang","Zhuang Liu","Laurens van der Maaten","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1608.06993","pdfUrl":"https://arxiv.org/pdf/1608.06993","codeUrl":null,"preprint":{"id":"1608.06993","submittedOn":"2016-08-25","url":"https://arxiv.org/abs/1608.06993"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1608.06993"},"citation":"@misc{classiccnnsdensenet2016,\n  title = {{Densely Connected Convolutional Networks}},\n  author = {Gao Huang and Zhuang Liu and Laurens van der Maaten and Kilian Q. Weinberger},\n  year = {2016},\n  eprint = {1608.06993},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1608.06993}\n}\n","citationPath":"/content/papers/citations/classic-cnns/densenet.6f28153b67f7.bib"},"thumbnail":{"src":"/content/papers/assets/4f073a0fd1fc5b7415ca19c48934d180f9043fddc08301819a50ceb1b0ebfc69.jpg","width":1184,"height":828},"locale":"en","path":"/content/papers/en/classic-cnns/densenet.9b1c0f86199d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"transition-layers-and-average-pooling","text":"Transition Layers and Average Pooling","depth":3},{"id":"growth-rate","text":"Growth Rate","depth":3},{"id":"bottleneck-layers","text":"Bottleneck Layers","depth":3},{"id":"compression","text":"Compression","depth":3},{"id":"implementation-details","text":"Implementation Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-results","text":"Experimental Results","depth":3},{"id":"compact-models","text":"Compact Models","depth":3},{"id":"feature-reuse","text":"Feature Reuse","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"background","topic":"Training deeper convolutional networks"}]}],"text":"Training very deep networks comes with a dilemma.\n\nIn a shallow network, signals flow smoothly forward and backward and training is fast, but its expressive power is limited. A deep network is more expressive, but gradients vanish, features computed early on get washed out by layer after layer of weights, and training time grows along with depth: a 152-layer ResNet takes weeks to train on ImageNet even with several GPUs.\n\nWhat if you trained a shallow network and tested a deep one?\n\nThat sounds self-contradictory. In March 2016, Gao Huang, Yu Sun, Daniel Sedra, and Kilian Q. Weinberger of Cornell University, together with Zhuang Liu of Tsinghua University, released this paper, later published at ECCV 2016. Their approach: build the network just as deep, but for every mini-batch during training, randomly remove some of the residual blocks, leaving only their shortcuts.\n\n## Defining the Problem\n\nThe authors list three problems that very deep networks run into:\n\n1. **Vanishing gradients**: as gradients propagate backward, they're multiplied by small weights again and again, so by the time they reach the early layers they've become tiny.\n2. **Diminishing forward features**: input features, or features computed by early layers, are multiplied again and again by randomly initialized weights until they're “washed out” further on, which makes it hard for later layers to find meaningful gradient directions.\n3. **Training takes too long**: forward and backward propagation time both grow linearly with depth.\n\nResNet and Highway Networks tackle the first two problems with shortcuts that let features and gradients bypass weight layers. The third problem remains: however deep the network is, every step has to compute that many layers.\n\nIs there a way to make a network shallower “on average” during training while keeping its full depth at test time?\n\n\n> [!TIP]\n> \n> For ResNet's residual blocks, you can start with our note:\n> \n> - **[[15.12] ResNet: Convolutional with 100 Layers](/blog/papers/classic-cnns/resnet)**\n\n\n## Solving the Problem\n\n### Letting Blocks Randomly Sit Out\n\nThe authors use ResNet's original residual block. The output of block $\\ell$ is:\n\n$$\nH_\\ell = \\mathrm{ReLU}\\big(f_\\ell(H_{\\ell-1}) + \\mathrm{id}(H_{\\ell-1})\\big)\n$$\n\n$f_\\ell$ is “convolution, BN, ReLU, convolution, BN,” and $\\mathrm{id}$ is the identity shortcut.\n\nStochastic depth adds just one Bernoulli random variable, $b_\\ell \\in \\{0, 1\\}$:\n\n$$\nH_\\ell = \\mathrm{ReLU}\\big(b_\\ell\\, f_\\ell(H_{\\ell-1}) + \\mathrm{id}(H_{\\ell-1})\\big)\n$$\n\nWhen $b_\\ell = 1$, this is the original residual block. When $b_\\ell = 0$, the residual branch disappears entirely, leaving $\\mathrm{ReLU}(H_{\\ell-1})$.\n\nThere's a small detail here: $H_{\\ell-1}$ is the output of the previous block's final ReLU, so it's never negative (the input to the first block is also the output of a “convolution, BN, ReLU” stack), which means $\\mathrm{ReLU}(H_{\\ell-1}) = H_{\\ell-1}$. An absent block is therefore exactly an identity mapping, and the signal passes to the next block untouched.\n\nThe probability that $b_\\ell = 1$ is written $p_\\ell$, which the authors call the **survival probability**. It's redrawn for every mini-batch, and a block drawn to sit out costs nothing at that step: no forward pass, no backward pass, and no weight update.\n\n### Early Layers Show Up More Often\n\nHow should each block's survival probability be set? The simplest choice is to make them all the same. The authors recommend a linear decay: $p_0 = 1$ at the input, $p_L$ for the last block, decreasing proportionally in between:\n\n$$\np_\\ell = 1 - \\frac{\\ell}{L}(1 - p_L)\n$$\n\n[![The top half is a row of bars labeled 1.0, 0.9, 0.8, 0.7, 0.6, and 0.5; their blue portions represent the survival probability and get shorter from left to right, and a small inset illustrates the two block states, active and inactive. The bottom half has a photo of a flower as the Input, followed by five residual blocks f1 to f5 with H1 to H4 between them; f3 and f5 are crossed out with red X's, meaning they're skipped this time](/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png)](/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png)\n\n*Fig. 2 from the paper, excerpted from [**page 5 of the v3 PDF**](https://arxiv.org/pdf/1603.09382v3#page=5). Linear decay with $p_0 = 1$ and $p_L = 0.5$; the input to the first block is treated as an always-present $H_0$.*\n\nThe authors' reasoning: early layers extract low-level features that every later layer uses, so they should show up more reliably. The experiments also show linear decay beating a uniform probability; and since results aren't very sensitive to $p_L$, they use $p_L = 0.5$ throughout the paper.\n\nTake the 110-layer ResNet commonly used in CIFAR experiments, which has $L = 54$ residual blocks:\n\n```python\nimport torch\n\nL, p_L = 54, 0.5                                  # ResNet-110: 54 residual blocks\np = torch.tensor([1 - l / L * (1 - p_L) for l in range(1, L + 1)])\nprint(\"Survival probabilities, first three blocks:\", [f\"{v:.3f}\" for v in p[:3]], \"and last three:\", [f\"{v:.3f}\" for v in p[-3:]])\nprint(\"Expected depth:\", round(p.sum().item(), 2), \"vs. formula (3L-1)/4 =\", (3 * L - 1) / 4)\n\ntorch.manual_seed(0)\nactive = torch.bernoulli(p.expand(10000, L)).sum(dim=1)   # sample 10000 mini-batches\nprint(\"Sampled: mean\", round(active.mean().item(), 2), \"blocks, ranging from\", int(active.min()), \"to\", int(active.max()))\nprint(\"Possible sub-networks: 2^54 =\", f\"{2 ** L:.3e}\")\n```\n\n```text\nSurvival probabilities, first three blocks: ['0.991', '0.981', '0.972'] and last three: ['0.519', '0.509', '0.500']\nExpected depth: 40.25 vs. formula (3L-1)/4 = 40.25\nSampled: mean 40.25 blocks, ranging from 28 to 51\nPossible sub-networks: 2^54 = 1.801e+16\n```\n\nDuring training, each step uses only about 40 blocks on average; at test time, all 54. The expected depth is $\\sum_\\ell p_\\ell = (3L-1)/4$, roughly $3L/4$, and the quarter that isn't computed is the training time saved. The sampled depth also varies a lot, anywhere from 28 to 51 blocks, so some updates go through especially short networks, which gives gradients a more direct path to the early layers.\n\n### Multiply by the Survival Probability at Test Time\n\nAt test time, every block has to be present so the full capacity of the network gets used. But during training, $f_\\ell$ appears in only a fraction $p_\\ell$ of the updates, and the next layer's weights are calibrated to that fraction. So at test time, each block's output is multiplied by its survival probability:\n\n$$\nH_\\ell^{\\text{Test}} = \\mathrm{ReLU}\\big(p_\\ell\\, f_\\ell(H_{\\ell-1}^{\\text{Test}}; W_\\ell) + H_{\\ell-1}^{\\text{Test}}\\big)\n$$\n\nLet's write a block and compare its outputs during training and testing. To keep things easy to see, the residual function here is a single linear layer, and we leave out the final ReLU for now:\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass StochasticDepthBlock(nn.Module):\n    def __init__(self, f, p):\n        super().__init__()\n        self.f, self.p = f, p                      # p: this block's survival probability\n\n    def forward(self, x):\n        if self.training:\n            if torch.rand(()) < self.p:            # survives: compute the residual as usual\n                return x + self.f(x)\n            return x                               # doesn't survive: the whole block becomes an identity mapping\n        return x + self.p * self.f(x)              # testing: multiply by the survival probability\n\ntorch.manual_seed(0)\nblock = StochasticDepthBlock(nn.Linear(4, 4), p=0.6)\nx = torch.randn(1, 4)\n\nblock.train()\nwith torch.no_grad():\n    outs = torch.stack([block(x) for _ in range(20000)])\nblock.eval()\nwith torch.no_grad():\n    test = block(x)\n\nprint(\"During training, the block produces\", torch.unique(outs.round(decimals=4), dim=0).shape[0], \"distinct outputs\")\nprint(\"Mean of training outputs:\", [f\"{v:.3f}\" for v in outs.mean(0).squeeze()])\nprint(\"Output at test time:     \", [f\"{v:.3f}\" for v in test.squeeze()])\n```\n\n```text\nDuring training, the block produces 2 distinct outputs\nMean of training outputs: ['-0.479', '0.711', '-1.017', '1.223']\nOutput at test time:      ['-0.479', '0.713', '-1.017', '1.223']\n```\n\nDuring training, this block has only two possible outputs: with the residual and without it. Their average over twenty thousand runs is almost identical to the test-time output, which multiplies by $p = 0.6$.\n\n\n> [!NOTE]\n> \n> Most libraries today move the correction to training time, the same way inverted dropout does: when a block survives, its residual is divided by the survival probability, so nothing needs to change at test time. In torchvision's `torchvision.ops.stochastic_depth`, for example, the parameter `p` is the probability of being “dropped,” and surviving inputs are divided by $1-p$. The two approaches have the same expected value; when reading code, pay attention to which probability `p` refers to.\n\n\n### Why Does It Work?\n\nThe authors give three reasons.\n\n- **Faster training**: skipped blocks aren't computed, so forward and backward propagation time follows the expected depth rather than the full depth.\n- **Stronger gradients**: the network is shorter during training, so gradients pass through fewer layers, and the early layers receive larger gradients.\n- **An implicit ensemble**: with 54 blocks, each either present or absent, there are $2^{54}$ combinations in all. Each mini-batch samples one weight-sharing sub-network to update, and at test time the formula above merges them into a single network. The record-breaking results in the ResNet paper came precisely from an ensemble of ResNets of different depths.\n\nThis ensemble view resembles dropout. The difference is that dropout makes the network **narrower**, while stochastic depth makes it **shallower**. The authors also note that dropout often becomes less effective when combined with batch normalization: they tried various dropout rates on a 110-layer ResNet with BN and saw almost no improvement, while stochastic depth still worked on the same network.\n\n## Discussion\n\n### Error Rates and Training Time\n\n| Method | CIFAR-10+ | CIFAR-100+ | SVHN | ImageNet |\n| --- | ---: | ---: | ---: | ---: |\n| Highway Network | 7.60 | 32.24 | – | – |\n| Gen. Pool | 6.05 | 28.02 | 1.69 | – |\n| ResNet, constant depth | 6.41 | 27.76 | 1.80 | 21.78 |\n| ResNet, stochastic depth | **5.25** | **24.98** | 1.75 | 21.98 |\n\n*Table: Selected rows excerpted from Table 1 of the paper, test error (%); “+” means standard data augmentation is used. “Constant depth” is the authors' own reproduction of He et al.'s experiments.*\n\nCIFAR-10 uses a 110-layer ResNet trained for 500 epochs. With constant depth, the error is 6.41%; adding stochastic depth brings it down to 5.25%, an 18% relative improvement. The training and network settings are exactly the same; the only difference is randomly skipping blocks during training. CIFAR-100 uses the same network and goes from 27.76% to 24.98%.\n\n[![Two sets of training curves: on the left, a 110-layer ResNet on CIFAR-10; on the right, CIFAR-100. The red line is the test error with constant depth, the blue line the test error with stochastic depth, and the two thin lines are their training losses. After the learning rate drops at epoch 250, the blue line is clearly below the red one; the left plot marks 6.41% and 5.25%, and the right marks 27.88% and 24.98%](/content/papers/assets/757bc4adb848efac2620e3a37c465e58ddc4a4c93e54f3b664bcea91cab64ffe.png)](/content/papers/assets/757bc4adb848efac2620e3a37c465e58ddc4a4c93e54f3b664bcea91cab64ffe.png)\n\n*Fig. 3 from the paper, excerpted from [**page 8 of the v3 PDF**](https://arxiv.org/pdf/1603.09382v3#page=8). Test error and training loss on CIFAR-10 (left) and CIFAR-100 (right); the dots mark the epoch with the lowest validation error.*\n\nAfter the first learning-rate drop, the constant-depth test error more or less stalls, while stochastic depth keeps going down. The authors also notice that the stochastic-depth test error fluctuates more, presumably because the depth keeps changing during training.\n\nWithout data augmentation, stochastic depth also gives about a 15% relative improvement on both CIFAR-10 and CIFAR-100: from 13.63% to 11.66%, and from 44.74% to 37.8%. SVHN uses a 152-layer network; with constant depth it starts to overfit as soon as the second phase begins, ending at 1.80% error, while stochastic depth gets 1.75%.\n\nTraining time compared:\n\n| | CIFAR-10+ | CIFAR-100+ | SVHN |\n| --- | ---: | ---: | ---: |\n| Constant depth | 20 h 42 min | 20 h 51 min | 33 h 43 min |\n| Stochastic depth | 15 h 7 min | 15 h 20 min | 25 h 33 min |\n\n*Table: Excerpted from Table 2 of the paper, with linear decay and $p_L = 0.5$.*\n\nEvery setting saves about a quarter of the time, consistent with the expected depth computed earlier.\n\n### Twelve Hundred Layers\n\nThe ResNet paper tried a 1202-layer network, which overfit, ending with 7.93% test error, worse than 110 layers. The authors redo this experiment with the same 1202-layer network, training for 300 epochs with a warm-up learning rate of 0.01 for the first 10 epochs.\n\nThe constant-depth 1202-layer network gets 6.67% error, again worse than the 110-layer network's 6.41%. With stochastic depth, the error is 4.91%: a 27% relative improvement over the constant-depth 1202-layer network, better than the stochastic-depth 110-layer network (5.25%) as well, and with no sign of overfitting. The authors say this is the first time a network with more than a thousand layers has reduced test error further than a shallower network.\n\n\n> [!TIP]\n> \n> That same month, the ResNet authors took a different route, switching the residual unit to pre-activation; their 1001-layer ResNet reaches 4.92% on CIFAR-10 (4.62% with a mini-batch size of 64):\n> \n> - **[[16.03] ResNet v2: Keep the Shortcut Clean](/blog/papers/classic-cnns/resnet-v2)**\n\n\n### No Win on ImageNet\n\nImageNet uses the 152-layer ResNet. Trained for 90 epochs with constant depth, it gets 23.06% error; with stochastic depth, it's at 23.38% at epoch 90, slightly worse. The authors observed that the stochastic-depth validation error was still clearly falling, and since stochastic depth saves 25% of the computation, training it for 30 more epochs takes about the same total time as 90 epochs at constant depth. With the extra training, stochastic depth reaches 21.98%; but constant depth was also run for 30 more epochs, and it reached 21.78%.\n\nSo in this setting, stochastic depth brings no accuracy benefit. The authors quote an anonymous reviewer: models for ImageNet are still in a different regime from those for CIFAR. The model complexity ImageNet calls for may far exceed a 152-layer ResNet, and stochastic depth may only show its effect with larger models.\n\n### Are the Gradients Really Stronger?\n\n[![A line chart with epochs on the horizontal axis (0 to 500) and the first convolutional layer's mean gradient magnitude on the vertical axis (in units of 10 to the minus 5). The red line is constant depth and the blue line is stochastic depth. Two dashed lines mark the learning-rate drops at epochs 250 and 375, where the gradients fall sharply; the blue line stays above the red throughout, and the gap is more pronounced after the learning-rate drops](/content/papers/assets/6e62eb722549bcdd22e104076055f0d6e70ad5e66c660f0e827fcfa67318c9a1.png)](/content/papers/assets/6e62eb722549bcdd22e104076055f0d6e70ad5e66c660f0e827fcfa67318c9a1.png)\n\n*Fig. 7 from the paper, excerpted from [**page 12 of the v3 PDF**](https://arxiv.org/pdf/1603.09382v3#page=12). Mean gradient magnitude per epoch of the first convolutional layer of the first residual block on CIFAR-10; the dashed lines mark when the learning rate is divided by 10.*\n\nThe two sharp drops come from the learning-rate schedule. Stochastic depth's gradients are larger throughout, and the gap widens after the learning-rate drops, which supports the authors' claim that it alleviates vanishing gradients.\n\n### What Should the Survival Probability Be?\n\n[![Left: test error of a 110-layer ResNet on CIFAR-10 as the survival probability p_L varies. The solid line is linear decay, the dashed line is a uniform probability, and the red dashed line is constant depth at 6.41%. The linear-decay curve is lowest around a p_L of 0.5, at about 5.25%, labeled with a 1.33x speedup; at a p_L of 0.2 it's still close to constant depth, labeled 1.67x; the uniform curve is mostly higher, dipping below linear decay only around a p_L of 0.8, and is labeled 2x at a p_L of 0.5. Right: a heat map with p_L on the vertical axis (1.0 to 0.5) and network depth on the horizontal axis (20 to 110 layers), with test errors in the cells; at 20 layers, a p_L between 0.9 and 0.7 beats constant depth and anything lower is worse, at 110 layers a p_L of 0.5 is best, and the lowest values lie roughly along the diagonal](/content/papers/assets/1f923643ee1fd74e75ef508bccb935e721a1b51f610ca3dcbac82168dc0573ad.png)](/content/papers/assets/1f923643ee1fd74e75ef508bccb935e721a1b51f610ca3dcbac82168dc0573ad.png)\n\n*Fig. 8 from the paper, excerpted from [**page 13 of the v3 PDF**](https://arxiv.org/pdf/1603.09382v3#page=13). Left: CIFAR-10 test error of a 110-layer ResNet for different values of $p_L$; right: test error across $p_L$ and network depth, where the $p_L = 1$ row is constant depth.*\n\n$p_L$ is the method's only hyperparameter. From the left plot:\n\n1. As long as $p_L$ is set reasonably, both rules beat constant depth.\n2. For most values of $p_L$, linear decay beats a uniform probability.\n3. Linear decay isn't very sensitive to $p_L$, giving good results for $p_L$ between 0.4 and 0.8.\n4. Even at $p_L = 0.2$, linear decay's error is on par with constant depth, while training time drops by 40%.\n\nThe heat map on the right adds depth. For a 20-layer network, skipping a little ($p_L$ between 0.9 and 0.7) beats constant depth, but skipping more makes it worse: at $p_L = 0.5$ the error rises from 8.31% to 9.39%. For a 110-layer network, $p_L = 0.5$ is best. The best-performing cells lie roughly along the diagonal: the deeper the network, the more aggressive the skipping it can handle. The network has to be deep enough for stochastic depth to clearly beat constant depth; for shallower networks, more conservative skipping still helps.\n\n## Conclusion\n\nStochastic depth is a training method that leaves the test-time network unchanged: randomly skip residual blocks during training, bringing the expected depth down to about three quarters, then combine all the blocks at test time using the survival probabilities. Experimentally, error rates fall on CIFAR and SVHN, training time drops by about a quarter, and the 1202-layer network no longer overfits. The 152-layer ResNet on ImageNet, however, shows no accuracy benefit.\n\nIn some libraries, it goes by the name DropPath. A few months later, Gao Huang, Zhuang Liu, and Kilian Q. Weinberger from this paper proposed DenseNet, whose introduction cites stochastic depth right at the start: if training still works when layers are randomly skipped, then many layers in a residual network actually contribute very little.\n\n\n> [!TIP]\n> \n> For DenseNet, see our note:\n> \n> - **[[16.08] DenseNet: Connecting Everything](/blog/papers/classic-cnns/densenet)**\n\n\nStochastic depth is easy to use: during training, multiply each residual branch by a random 0 or 1; at test time, use the survival probability instead. Most libraries today divide by the survival probability during training instead, like torchvision's `stochastic_depth`, so nothing needs to change at test time.\n\nBack then, the authors guessed that stochastic depth might only pay off on ImageNet with larger models. Later, the training recipes of large models such as [ConvNeXt](/blog/papers/classic-cnns/convnext) and [DINOv2](/blog/papers/contrastive-learning/dinov2) did indeed include it.\n","imageSizes":{"/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png":{"width":963,"height":339},"/content/papers/assets/757bc4adb848efac2620e3a37c465e58ddc4a4c93e54f3b664bcea91cab64ffe.png":{"width":965,"height":372},"/content/papers/assets/6e62eb722549bcdd22e104076055f0d6e70ad5e66c660f0e827fcfa67318c9a1.png":{"width":756,"height":345},"/content/papers/assets/1f923643ee1fd74e75ef508bccb935e721a1b51f610ca3dcbac82168dc0573ad.png":{"width":973,"height":422}}}