{"notes":[{"id":"classic-cnns/wide-resnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Wide ResNet","subtitle":"Wider, Not Deeper","author":"Z. Yuan","paperDate":"2016-05-23","publishedOn":"2024-02-14","world":"external/wide-resnet","paperTitle":"Wide Residual Networks","paperUrl":"https://arxiv.org/abs/1605.07146","bibliography":{"title":"Wide Residual Networks","authors":["Sergey Zagoruyko","Nikos Komodakis"],"url":"https://arxiv.org/abs/1605.07146","pdfUrl":"https://arxiv.org/pdf/1605.07146v4","codeUrl":null,"preprint":{"id":"1605.07146","submittedOn":"2016-05-23","url":"https://arxiv.org/abs/1605.07146"},"publication":{"venue":"BMVC","year":2016,"kind":"conference","url":"https://doi.org/10.5244/C.30.87"},"citation":"@misc{classiccnnswideresnet2016,\n  title = {{Wide Residual Networks}},\n  author = {Sergey Zagoruyko and Nikos Komodakis},\n  year = {2016},\n  eprint = {1605.07146},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1605.07146}\n}\n","citationPath":"/content/papers/citations/classic-cnns/wide-resnet.2e42de74f658.bib"},"thumbnail":{"src":"/content/papers/assets/eb123078c88654918b04d17b5ebccd1009fc7fbc33305ebd9ba853f2ad2cc024.png","width":1400,"height":500},"locale":"en","path":"/content/papers/en/classic-cnns/wide-resnet.5ef1b2569375.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"resnets-keep-getting-thinner","text":"ResNets Keep Getting Thinner","depth":3},{"id":"the-shortcut-is-too-easy","text":"The Shortcut Is Too Easy","depth":3},{"id":"thin-networks-arent-fast-either","text":"Thin Networks Aren't Fast Either","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-can-be-changed","text":"What Can Be Changed","depth":3},{"id":"wrn-n-k","text":"WRN-n-k","depth":3},{"id":"what-goes-inside-a-block","text":"What Goes Inside a Block","depth":3},{"id":"putting-dropout-inside-the-block","text":"Putting Dropout Inside the Block","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"splitting-between-width-and-depth","text":"Splitting Between Width and Depth","depth":3},{"id":"taking-on-the-thousand-layer-resnet","text":"Taking On the Thousand-Layer ResNet","depth":3},{"id":"wide-networks-run-faster","text":"Wide Networks Run Faster","depth":3},{"id":"the-effect-of-dropout","text":"The Effect of Dropout","depth":3},{"id":"on-imagenet-widen-the-middle-layer","text":"On ImageNet, Widen the Middle Layer","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""},{"id":"classic-cnns/stochastic-depth","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"Keep the Shortcut Clean","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2024-01-11","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":{"venue":"ECCV","year":2016,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-46493-0_38"},"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"en","path":"/content/papers/en/classic-cnns/resnet-v2.1f8c5aedb2e0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"two-identity-mappings","text":"Two Identity Mappings","depth":3},{"id":"what-if-the-shortcut-carries-something-extra","text":"What If the Shortcut Carries Something Extra?","depth":3},{"id":"where-should-the-after-addition-relu-go","text":"Where Should the After-Addition ReLU Go?","depth":3},{"id":"a-different-angle-pre-activation","text":"A Different Angle: Pre-Activation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-deeper-the-wider-the-gap","text":"The Deeper, the Wider the Gap","depth":3},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]},{"id":"classic-cnns/stochastic-depth","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Stochastic Depth","subtitle":"This Layer Is Taking the Day Off","author":"Z. Yuan","paperDate":"2016-03-30","publishedOn":"2024-01-24","world":"external/stochastic-depth","paperTitle":"Deep Networks with Stochastic Depth","paperUrl":"https://arxiv.org/abs/1603.09382","bibliography":{"title":"Deep Networks with Stochastic Depth","authors":["Gao Huang","Yu Sun","Zhuang Liu","Daniel Sedra","Kilian Weinberger"],"url":"https://arxiv.org/abs/1603.09382","pdfUrl":"https://arxiv.org/pdf/1603.09382v3","codeUrl":null,"preprint":{"id":"1603.09382","submittedOn":"2016-03-30","url":"https://arxiv.org/abs/1603.09382"},"publication":{"venue":"ECCV","year":2016,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-46493-0_39"},"citation":"@misc{classiccnnsstochasticdepth2016,\n  title = {{Deep Networks with Stochastic Depth}},\n  author = {Gao Huang and Yu Sun and Zhuang Liu and Daniel Sedra and Kilian Weinberger},\n  year = {2016},\n  eprint = {1603.09382},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.09382}\n}\n","citationPath":"/content/papers/citations/classic-cnns/stochastic-depth.3716109ef477.bib"},"thumbnail":{"src":"/content/papers/assets/620e6ddf909f1f405e38b6055bd732ae4c4b7c208f147e00b870a56642020c02.png","width":963,"height":339},"locale":"en","path":"/content/papers/en/classic-cnns/stochastic-depth.71fd41fe9758.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"letting-blocks-randomly-sit-out","text":"Letting Blocks Randomly Sit Out","depth":3},{"id":"early-layers-show-up-more-often","text":"Early Layers Show Up More Often","depth":3},{"id":"multiply-by-the-survival-probability-at-test-time","text":"Multiply by the Survival Probability at Test Time","depth":3},{"id":"why-does-it-work","text":"Why Does It Work?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"error-rates-and-training-time","text":"Error Rates and Training Time","depth":3},{"id":"twelve-hundred-layers","text":"Twelve Hundred Layers","depth":3},{"id":"no-win-on-imagenet","text":"No Win on ImageNet","depth":3},{"id":"are-the-gradients-really-stronger","text":"Are the Gradients Really Stronger?","depth":3},{"id":"what-should-the-survival-probability-be","text":"What Should the Survival Probability Be?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""},{"id":"classic-cnns/densenet","relation":"referenced","topic":""}]}],"text":"In the first half of 2016, ResNets kept getting deeper and deeper.\n\nThe ResNet authors switched the residual unit to pre-activation and trained a 1001-layer network on CIFAR-10; around the same time, Stochastic Depth stacked a network up to 1202 layers.\n\n\n> [!TIP]\n> \n> We have notes on both, so you can start there:\n> \n> - **[[16.03] ResNet v2: Keep the Shortcut Clean](/blog/papers/classic-cnns/resnet-v2)**\n> - **[[16.03] Stochastic Depth: This Layer Is Taking the Day Off](/blog/papers/classic-cnns/stochastic-depth)**\n\n\nMore layers did give better results; they just kept getting more expensive. Going from 110 to 1001 layers, the pre-activation ResNet's CIFAR-10 error fell from 6.37% to 4.92%. The layer count went up ninefold, the error came down by less than one and a half percentage points, and training time stretched out along with it.\n\nSergey Zagoruyko and Nikos Komodakis of École des Ponts ParisTech in France thought about it from another angle. Does a ResNet really have to be this thin?\n\nThey made the network wider and shallower.\n\n## Defining the Problem\n\n### ResNets Keep Getting Thinner\n\nThe ResNet authors favored thin, deep networks, trading the parameters they saved for more layers. The later bottleneck block first reduces dimensionality with a $1\\times1$ convolution, runs the $3\\times3$ convolution, and then expands back up, making each block thinner still.\n\nTake the ResNet used on CIFAR. Its three stages have just 16, 32, and 64 channels. VGG and Inception, which came before it, all had far more channels than that.\n\n### The Shortcut Is Too Easy\n\nThe authors point out that the identity shortcut, which lets ResNets train to a thousand layers and beyond, is also their weakness.\n\nGradients can travel all the way back along the shortcut, and nothing forces them through the weights of the residual branch. After training, perhaps only a few blocks have learned anything useful, or many blocks each contribute just a little. The authors call this phenomenon diminishing feature reuse.\n\nThe authors cite Stochastic Depth as evidence. Randomly switching off entire residual blocks during training actually makes the network better, which, as the authors see it, shows that many blocks weren't doing much in the first place.\n\n### Thin Networks Aren't Fast Either\n\nGPUs are good at processing large tensors in one go, but a thin, deep network is a long chain of small convolutions that have to be computed one layer after another. As the authors note in the abstract, each small gain in accuracy requires nearly doubling the number of layers, so training gets slower and slower.\n\nSo the authors wanted to work out whether the same parameters are better used to make the network deeper or wider. Just how wide should a ResNet be?\n\n## Solving the Problem\n\n### What Can Be Changed\n\nThe authors start from ResNet v2's pre-activation block, where BN and ReLU come before every convolution. There are roughly three ways to make a residual block more expressive:\n\n1. Put more convolutional layers in the block.\n2. Widen the convolutions with more channels.\n3. Make the kernels larger.\n\nThe third is ruled out up front, since work on VGG and the Inception family has already shown how effective small $3\\times3$ kernels are. For the remaining two, the authors define one factor each. The **deepening factor $l$** is the number of convolutional layers in a block, and the **widening factor $k$** is how many times the channel count is multiplied. The original block is $l=2$, $k=1$.\n\n[![Diagrams of four residual blocks. In each, the gray shortcut is on the left and the residual branch on the right, and the two are added at the bottom. (a) basic: two conv3x3; (b) bottleneck: conv1x1, conv3x3, conv1x1, with the middle conv3x3 narrower; (c) basic-wide: two wider conv3x3; (d) wide-dropout: a dropout layer sandwiched between two wider conv3x3](/content/papers/assets/eb123078c88654918b04d17b5ebccd1009fc7fbc33305ebd9ba853f2ad2cc024.png)](/content/papers/assets/eb123078c88654918b04d17b5ebccd1009fc7fbc33305ebd9ba853f2ad2cc024.png)\n\n*Figure 1 from the paper, excerpted from [**page 2 of the v4 PDF**](https://arxiv.org/pdf/1605.07146v4#page=2). The four kinds of residual block used in the paper. Every convolution is preceded by BN and ReLU, which the figure omits.*\n\nThe width of each box in the figure represents the channel count. (a) is the original basic block; (c) has the same structure, only with both convolutions widened. The bottleneck in (b) is the tool ResNet uses to make networks thinner, the opposite of the widening this paper studies, so the authors leave it out and build everything from the basic block. (d) adds dropout between the two convolutions; we'll come back to it later.\n\n### WRN-n-k\n\nThe network's skeleton follows ResNet v2's design for CIFAR, with the widening factor $k$ as the only addition:\n\n| Group | Output size | Contents |\n| --- | --- | --- |\n| conv1 | $32\\times32$ | $3\\times3$ convolution, 16 channels |\n| conv2 | $32\\times32$ | $N$ blocks, each with two $3\\times3$ convolutional layers, $16k$ channels |\n| conv3 | $16\\times16$ | $N$ blocks, $32k$ channels |\n| conv4 | $8\\times8$ | $N$ blocks, $64k$ channels |\n| avg-pool | $1\\times1$ | $8\\times8$ average pooling, followed by the classification layer |\n\n*Table: Compiled from Table 1 of the paper. The first layer of conv3 and of conv4 halves the feature map size; $k=1$ is the original pre-activation ResNet.*\n\nThe authors denote a network as **WRN-$n$-$k$**, where $n$ is the total number of convolutional layers and $k$ is the widening factor. The official code requires $n=6N+4$, so WRN-28-10 is a network with $N=4$ blocks per group and channel counts multiplied by 10.\n\n$k$ affects the parameter count quadratically. A $3\\times3$ convolution whose input and output both have $C$ channels has $9C^2$ weights, so doubling the channels quadruples the weights. Let's build a few networks following the table above and count their parameters:\n\n```python\nimport torch.nn as nn\n\nclass Block(nn.Module):\n    \"\"\"BN-ReLU-conv3×3-BN-ReLU-conv3×3; the shortcut uses a 1×1 convolution when the shape changes\"\"\"\n    def __init__(self, cin, cout, stride):\n        super().__init__()\n        self.body = nn.Sequential(\n            nn.BatchNorm2d(cin), nn.ReLU(), nn.Conv2d(cin, cout, 3, stride, 1, bias=False),\n            nn.BatchNorm2d(cout), nn.ReLU(), nn.Conv2d(cout, cout, 3, 1, 1, bias=False))\n        self.shortcut = (nn.Identity() if cin == cout and stride == 1\n                         else nn.Conv2d(cin, cout, 1, stride, bias=False))\n\n    def forward(self, x):\n        return self.shortcut(x) + self.body(x)\n\ndef wrn(n, k, num_classes=10):\n    N = (n - 4) // 6                       # blocks per group\n    widths = [16 * k, 32 * k, 64 * k]\n    layers, cin = [nn.Conv2d(3, 16, 3, 1, 1, bias=False)], 16\n    for width, stride in zip(widths, (1, 2, 2)):\n        for i in range(N):\n            layers.append(Block(cin, width, stride if i == 0 else 1))\n            cin = width\n    layers += [nn.BatchNorm2d(cin), nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(cin, num_classes)]\n    return nn.Sequential(*layers)\n\nfor n, k in [(40, 1), (40, 2), (40, 4), (16, 8), (22, 8), (28, 10)]:\n    params = sum(p.numel() for p in wrn(n, k).parameters())\n    print(f\"WRN-{n}-{k}: {(n - 4) // 6} blocks per group, channels {16*k}/{32*k}/{64*k}, {params / 1e6:.1f}M parameters\")\n```\n\n```text\nWRN-40-1: 6 blocks per group, channels 16/32/64, 0.6M parameters\nWRN-40-2: 6 blocks per group, channels 32/64/128, 2.2M parameters\nWRN-40-4: 6 blocks per group, channels 64/128/256, 8.9M parameters\nWRN-16-8: 2 blocks per group, channels 128/256/512, 11.0M parameters\nWRN-22-8: 3 blocks per group, channels 128/256/512, 17.2M parameters\nWRN-28-10: 4 blocks per group, channels 160/320/640, 36.5M parameters\n```\n\nThese numbers match the parameter counts in the paper's tables. At the same 40 layers, as $k$ goes from 1 to 2 to 4, the parameters go from 0.6M to 2.2M to 8.9M, roughly four times as many at each step. Looked at the other way, WRN-16-8 has only 16 layers, yet it already has more parameters than the 40-layer WRN-40-4.\n\n### What Goes Inside a Block\n\nShould some $1\\times1$ convolutions be mixed in, as in NiN? The authors write a block's structure as $B(M)$, where $M$ lists the kernel sizes of the block's convolutional layers. For example, $B(3,3)$ is the original basic block, and $B(3,1,3)$ inserts an extra $1\\times1$ layer between the two $3\\times3$ layers.\n\nThese experiments all use $k=2$, with the depth adjusted to keep the parameter counts similar, and each configuration is run 5 times on CIFAR-10, taking the median:\n\n| Block | Depth | Parameters | One epoch (s) | CIFAR-10 error |\n| --- | ---: | ---: | ---: | ---: |\n| $B(1,3,1)$ | 40 | 1.4M | 85.8 | 6.06% |\n| $B(3,1)$ | 40 | 1.2M | 67.5 | 5.78% |\n| $B(1,3)$ | 40 | 1.3M | 72.2 | 6.42% |\n| $B(3,1,1)$ | 40 | 1.3M | 82.2 | 5.86% |\n| **$B(3,3)$** | 28 | 1.5M | 67.5 | **5.73%** |\n| $B(3,1,3)$ | 22 | 1.1M | 59.9 | 5.78% |\n\n*Table: Excerpted from Table 2 of the paper.*\n\n$B(3,3)$ is the best, but only by a small margin; $B(3,1)$ and $B(3,1,3)$ have fewer parameters and score almost the same. From this, the authors judge that blocks with similar parameter counts perform about the same, and from then on they use $3\\times3$ convolutions throughout.\n\nHow many layers should go in a block? The authors fix WRN-40-2's total layer count and parameter count (2.2M) and vary only the number of layers per block, $l$:\n\n| $l$ | 1 | 2 | 3 | 4 |\n| --- | ---: | ---: | ---: | ---: |\n| CIFAR-10 error | 6.69% | **5.43%** | 5.65% | 5.93% |\n\n*Table: Excerpted from Table 3 of the paper.*\n\nWith the total layer count fixed, a larger $l$ means fewer blocks, and therefore fewer residual connections. The authors speculate that $l=3$ and $l=4$ are worse because the network, with fewer residual connections, is harder to optimize; $l=1$, with a single layer per block, is clearly worse still. Two layers is just right, which is the original $B(3,3)$.\n\nAfter trying all of that, the block's internals end up right back where they started, and the only thing left to change is the width $k$.\n\n### Putting Dropout Inside the Block\n\nWidening makes the parameter count balloon, which calls for regularization. BN has a regularizing effect of its own, but it usually needs heavy data augmentation alongside it, and the authors didn't want to rely on that.\n\nThey add a dropout layer between the block's two convolutions, after the ReLU, as in Figure 1(d). In code, that's one extra `nn.Dropout` after the second ReLU of the `Block` above:\n\n```python\n        self.body = nn.Sequential(\n            nn.BatchNorm2d(cin), nn.ReLU(), nn.Conv2d(cin, cout, 3, stride, 1, bias=False),\n            nn.BatchNorm2d(cout), nn.ReLU(), nn.Dropout(0.3),   # between the two convolutions\n            nn.Conv2d(cout, cout, 3, 1, 1, bias=False))\n```\n\nResNet v2 had also tried dropout in the residual block, but on the shortcut, and results got worse. Here the dropout touches only the residual branch, and the shortcut stays clean. The dropout probability is chosen by cross-validation, 0.3 for CIFAR and 0.4 for SVHN.\n\n## Discussion\n\nApart from ImageNet, every experiment uses the pre-activation ResNet as the baseline. CIFAR-10 and CIFAR-100 consist of small $32\\times32$ color images, each with 50,000 for training and 10,000 for testing, in 10 and 100 classes, respectively; the only data augmentation is horizontal flips and random crops. SVHN contains street-view house numbers, about 600,000 images, with no augmentation at all.\n\n### Splitting Between Width and Depth\n\n| Depth | $k$ | Parameters | CIFAR-10 | CIFAR-100 |\n| ---: | ---: | ---: | ---: | ---: |\n| 40 | 1 | 0.6M | 6.85% | 30.89% |\n| 40 | 2 | 2.2M | 5.33% | 26.04% |\n| 40 | 4 | 8.9M | 4.97% | 22.89% |\n| 40 | 8 | 35.7M | 4.66% | – |\n| 28 | 10 | 36.5M | **4.17%** | 20.50% |\n| 28 | 12 | 52.5M | 4.33% | **20.43%** |\n| 22 | 8 | 17.2M | 4.38% | 21.22% |\n| 22 | 10 | 26.8M | 4.44% | 20.75% |\n| 16 | 8 | 11.0M | 4.81% | 22.07% |\n| 16 | 10 | 17.1M | 4.56% | 21.59% |\n\n*Table: Excerpted from Table 4 of the paper. CIFAR test error, with images preprocessed by ZCA whitening.*\n\nFirst, hold the depth fixed. For the 40-, 22-, and 16-layer networks, a larger $k$ generally means a lower error. The only small exception is the 22-layer network going from $k=8$ to $k=10$, where the CIFAR-10 error rises slightly from 4.38% to 4.44% while the CIFAR-100 error keeps falling.\n\nNow hold the width fixed. At $k=8$, going from 16 to 22 layers helps, but going further to 40 layers actually hurts, and WRN-40-8 loses to WRN-22-8. WRN-40-8 has 35.7M parameters, and the authors' view is that when depth and width grow together, once the parameter count passes a certain point, stronger regularization is needed.\n\n### Taking On the Thousand-Layer ResNet\n\nNext comes a head-to-head comparison with the strongest thin networks of the time. These experiments switch to plain mean/std normalization, matching the preprocessing in the ResNet papers, and none of them use dropout:\n\n| Network | Depth-$k$ | Parameters | CIFAR-10 | CIFAR-100 |\n| --- | --- | ---: | ---: | ---: |\n| Original ResNet | 110 | 1.7M | 6.43% | 25.16% |\n| Original ResNet | 1202 | 10.2M | 7.93% | 27.82% |\n| Stochastic Depth | 1202 | 10.2M | 4.91% | – |\n| pre-act ResNet | 164 | 1.7M | 5.46% | 24.33% |\n| pre-act ResNet | 1001 | 10.2M | 4.92% (4.64%) | 22.71% |\n| WRN | 40-4 | 8.9M | 4.53% | 21.18% |\n| WRN | 16-8 | 11.0M | 4.27% | 20.43% |\n| WRN | 28-10 | 36.5M | **4.00%** | **19.25%** |\n\n*Table: Excerpted from Table 5 of the paper. The number in parentheses is ResNet v2's result with a batch size of 64; all other numbers use a batch size of 128.*\n\nWRN-40-4 and ResNet-1001 have similar parameter counts (8.9M and 10.2M), yet with a twenty-fifth of the layers, WRN-40-4 wins on both datasets. From this, the authors conclude that at this parameter scale, depth brings no more regularization than width does.\n\nWRN-28-10 has 3.6 times as many parameters as ResNet-1001. The common belief at the time was that depth has a regularizing effect while width invites overfitting, and by that logic it should overfit. In fact, at the same batch size, it wins by 0.92 percentage points on CIFAR-10 and 3.46 percentage points on CIFAR-100.\n\n### Wide Networks Run Faster\n\n[![Bar chart with time (ms) on the vertical axis. The two blue bars on the left are thin networks: 164 layers at 85 ms and a thousand layers at 512 ms. The three red bars on the right are wide networks: WRN-40-4 at 68 ms, WRN-16-10 at 164 ms, and WRN-28-10 at 312 ms. Each bar is labeled with its CIFAR-10 test error](/content/papers/assets/a243b50be60b82d957c92fa85eab5061afe10a86de40a8855c992996066803b0.png)](/content/papers/assets/a243b50be60b82d957c92fa85eab5061afe10a86de40a8855c992996066803b0.png)\n\n*Figure 4 from the paper, excerpted from [**page 12 of the v4 PDF**](https://arxiv.org/pdf/1605.07146v4#page=12). Time for one forward and backward pass at a batch size of 32 (Titan X, cuDNN v5). The horizontal axis shows the depth, or depth-$k$.*\n\nBlue is thin and red is wide. The thousand-layer thin ResNet takes 512 ms per batch (when a thousand layers have to queue up, nothing moves fast), while WRN-40-4, with about the same accuracy, needs only 68 ms, more than seven times faster. The most accurate one, WRN-28-10, has several times as many parameters and is still 1.6 times faster than the thousand-layer network.\n\nThe gap comes from the GPU behavior mentioned earlier. A wide convolutional layer processes a big block of tensor at once, keeping the GPU's parallel compute fully busy; a thin network is a thousand small convolutions lined up in a row, with each layer waiting for the previous one to finish.\n\n### The Effect of Dropout\n\n| Depth | $k$ | Dropout | CIFAR-10 | CIFAR-100 | SVHN |\n| ---: | ---: | :---: | ---: | ---: | ---: |\n| 16 | 4 | | 5.02% | 24.03% | 1.85% |\n| 16 | 4 | ✓ | 5.24% | 23.91% | 1.64% |\n| 28 | 10 | | 4.00% | 19.25% | – |\n| 28 | 10 | ✓ | **3.89%** | **18.85%** | – |\n| 52 | 1 | | 6.43% | 29.89% | 2.08% |\n| 52 | 1 | ✓ | 6.28% | 29.78% | 1.70% |\n\n*Table: Excerpted from Table 6 of the paper. Mean/std preprocessing; the CIFAR numbers are medians of 5 runs.*\n\nOn CIFAR, the gains are small. With dropout, WRN-28-10's error drops by 0.11 percentage points on CIFAR-10 and 0.4 percentage points on CIFAR-100; WRN-16-4, which has fewer parameters, even does slightly worse on CIFAR-10.\n\nOn SVHN, the effect is clear, with both the wide and the thin networks improving by 0.2 to 0.4 percentage points. SVHN gets no data augmentation at all, and the authors think BN overfits here; without dropout, the training loss falls to very low values.\n\nThe authors also noticed that after the first learning-rate drop, the loss and validation error would suddenly rise together and oscillate at high values until the next learning-rate drop. They traced the cause to weight decay, but lowering the weight decay noticeably hurt accuracy. In most cases, dropout partially removes this effect.\n\nThe authors also list their best single-model results on each dataset. WRN-40-10 with dropout reaches 3.8% on CIFAR-10 and 18.3% on CIFAR-100, and WRN-16-8 with dropout reaches 1.54% on SVHN.\n\n### On ImageNet, Widen the Middle Layer\n\nOn ImageNet, the authors first widen basic-block networks such as ResNet-18 and ResNet-34 by 1.5 to 3 times. Widening helps here too, and networks with similar parameter counts reach similar accuracy even when one is twice as deep as the other. But these networks have anywhere from tens of millions to over a hundred million parameters, and they still can't match bottleneck networks with far fewer. For example, WRN-34 widened twofold has 86.0M parameters and a top-1 error of 23.39%, while ResNet-101 gets 22.44% with just 44.5M parameters.\n\nThe authors suspect that either the bottleneck is simply better suited to ImageNet, or this task needs a deeper network. So they start from ResNet-50 instead and widen only the $3\\times3$ convolution in the middle of the bottleneck by a factor of two, giving WRN-50-2-bottleneck:\n\n| Network | Top-1 error | Top-5 error | Parameters | Time per batch (ms) |\n| --- | ---: | ---: | ---: | ---: |\n| ResNet-50 | 24.01% | 7.02% | 25.6M | 49 |\n| ResNet-101 | 22.44% | 6.21% | 44.5M | 82 |\n| ResNet-152 | 22.16% | 6.16% | 60.2M | 115 |\n| **WRN-50-2-bottleneck** | **21.9%** | **6.03%** | 68.9M | 93 |\n| pre-ResNet-200 | 21.66% | 5.79% | 64.7M | 154 |\n\n*Table: Excerpted from Table 8 of the paper. ILSVRC-2012 validation set, single crop; times are for a batch size of 16.*\n\nWith only 50 layers, WRN-50-2 beats ResNet-152 and is faster too; compared with the 200-layer pre-activation ResNet, its error is slightly higher, but it needs only 93 ms per batch versus 154 ms. The authors also say frankly that larger bottleneck networks need an 8-GPU machine to train, so they didn't try any.\n\nThis network is `wide_resnet50_2` in torchvision. Let's pull out the last block of it and of ResNet-50 and compare them:\n\n```python\nfrom torchvision.models import resnet50, wide_resnet50_2\n\nfor name, build in [(\"ResNet-50\", resnet50), (\"Wide ResNet-50-2\", wide_resnet50_2)]:\n    model = build(weights=None)\n    last = model.layer4[-1]\n    widths = (last.conv1.in_channels, last.conv2.in_channels, last.conv3.out_channels)\n    params = sum(p.numel() for p in model.parameters())\n    print(f\"{name}: last block {widths[0]}-{widths[1]}-{widths[2]}, {params / 1e6:.1f}M parameters\")\n```\n\n```text\nResNet-50: last block 2048-512-2048, 25.6M parameters\nWide ResNet-50-2: last block 2048-1024-2048, 68.9M parameters\n```\n\nThe block's input and output still have 2048 channels; only the middle layer goes from 512 to 1024. The parameter counts also match the paper's table.\n\nThe authors also entered the COCO 2016 object detection challenge with WRN-34-2, a 34-layer network widened twofold, reaching 35.2 mAP as a single model and beating models built on ResNet-152 and Inception-v4 backbones.\n\n## Conclusion\n\nBack to the question from the start. Does a ResNet really have to be this thin? This paper's answer is no. On CIFAR, the same parameters do more good as width than as depth; a 16-layer WRN already beats the thousand-layer ResNet and trains several times faster. Inside that thousand-layer ResNet, it looks like quite a few layers were slacking off.\n\nThis conclusion comes mainly from CIFAR. On ImageNet, the authors only went up to 50 layers, for the practical reason that larger networks need an 8-GPU machine to train.\n\nWRN later became one of the most common backbones on CIFAR; the semi-supervised [FixMatch](https://arxiv.org/abs/2001.07685), for example, uses WRN-28-2. Next time you see that name in a paper, you'll know it's a 28-layer network widened twofold.\n","imageSizes":{"/content/papers/assets/eb123078c88654918b04d17b5ebccd1009fc7fbc33305ebd9ba853f2ad2cc024.png":{"width":1400,"height":500},"/content/papers/assets/a243b50be60b82d957c92fa85eab5061afe10a86de40a8855c992996066803b0.png":{"width":459,"height":542}}}