{"notes":[{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"Keep the Shortcut Clean","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"en","path":"/content/papers/en/classic-cnns/resnet-v2.1b56c20d8b4c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"two-identity-mappings","text":"Two Identity Mappings","depth":3},{"id":"what-if-the-shortcut-carries-something-extra","text":"What If the Shortcut Carries Something Extra?","depth":3},{"id":"where-should-the-after-addition-relu-go","text":"Where Should the After-Addition ReLU Go?","depth":3},{"id":"a-different-angle-pre-activation","text":"A Different Angle: Pre-Activation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-deeper-the-wider-the-gap","text":"The Deeper, the Wider the Gap","depth":3},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"Convolutional with 100 Layers","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"en","path":"/content/papers/en/classic-cnns/resnet.b65375cb9846.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-is-residual","text":"What is Residual?","depth":3},{"id":"residual-learning","text":"Residual Learning","depth":3},{"id":"residual-module","text":"Residual Module","depth":3},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"architectures-and-computation","text":"Architectures and Computation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"deeper-bottleneck-architectures","text":"Deeper Bottleneck Architectures","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"Training deeper convolutional networks"},{"id":"classic-cnns/densenet","relation":"next","topic":"Training deeper convolutional networks"}]},{"id":"classic-cnns/highway-networks","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Highway Networks","subtitle":"On the Eve of Residuals","author":"Z. Yuan","paperDate":"2015-05-03","publishedOn":"2025-11-03","world":"external/highway-networks","paperTitle":"Highway Networks","paperUrl":"https://arxiv.org/abs/1505.00387","bibliography":{"title":"Highway Networks","authors":["Rupesh Kumar Srivastava","Klaus Greff","Jürgen Schmidhuber"],"url":"https://arxiv.org/abs/1505.00387","pdfUrl":"https://arxiv.org/pdf/1505.00387v2","codeUrl":null,"preprint":{"id":"1505.00387","submittedOn":"2015-05-03","url":"https://arxiv.org/abs/1505.00387"},"publication":null,"citation":"@misc{classiccnnshighwaynetworks2015,\n  title = {{Highway Networks}},\n  author = {Rupesh Kumar Srivastava and Klaus Greff and Jürgen Schmidhuber},\n  year = {2015},\n  eprint = {1505.00387},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1505.00387}\n}\n","citationPath":"/content/papers/citations/classic-cnns/highway-networks.f7d3ef9547ef.bib"},"thumbnail":{"src":"/content/papers/assets/2cef37bd2a2388f57a76965431810cb86f976db701ba1647b75c81c9a59ac779.png","width":1344,"height":360},"locale":"en","path":"/content/papers/en/classic-cnns/highway-networks.7580f83fc759.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"adding-two-gates","text":"Adding Two Gates","depth":3},{"id":"lean-toward-pass-straight-through-at-first","text":"Lean Toward “Pass Straight Through” at First","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-deeper-the-easier-to-train","text":"The Deeper, the Easier to Train?","depth":3},{"id":"easy-to-train-but-what-about-accuracy","text":"Easy to Train, but What About Accuracy?","depth":3},{"id":"what-did-the-gates-learn","text":"What Did the Gates Learn?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/inception-v4","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v4","subtitle":"Much Faster, a Little More Accurate","author":"Z. Yuan","paperDate":"2016-02-23","publishedOn":"2026-01-27","world":"external/inception-v4","paperTitle":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","paperUrl":"https://arxiv.org/abs/1602.07261","bibliography":{"title":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","authors":["Christian Szegedy","Sergey Ioffe","Vincent Vanhoucke","Alex Alemi"],"url":"https://arxiv.org/abs/1602.07261","pdfUrl":"https://arxiv.org/pdf/1602.07261v2","codeUrl":null,"preprint":{"id":"1602.07261","submittedOn":"2016-02-23","url":"https://arxiv.org/abs/1602.07261"},"publication":null,"citation":"@misc{classiccnnsinceptionv42016,\n  title = {{Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning}},\n  author = {Christian Szegedy and Sergey Ioffe and Vincent Vanhoucke and Alex Alemi},\n  year = {2016},\n  eprint = {1602.07261},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1602.07261}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v4.ed8e0294ae0b.bib"},"thumbnail":{"src":"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png","width":666,"height":656},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v4.f625ed245ee7.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"dropping-the-baggage-inception-v4","text":"Dropping the Baggage: Inception-v4","depth":3},{"id":"the-entrance","text":"The Entrance","depth":3},{"id":"wiring-residuals-into-inception","text":"Wiring Residuals into Inception","depth":3},{"id":"scaling-down-the-residuals-to-steady-training","text":"Scaling Down the Residuals to Steady Training","depth":3},{"id":"training-setup","text":"Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"residuals-make-training-much-faster","text":"Residuals Make Training Much Faster","depth":3},{"id":"single-model-results","text":"Single-Model Results","depth":3},{"id":"ensembles","text":"Ensembles","depth":3},{"id":"are-residual-connections-necessary","text":"Are Residual Connections “Necessary”?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""}]}],"text":"ResNet made 152-layer networks trainable and won ILSVRC 2015. On CIFAR-10, though, the original paper's 1202-layer ResNet had an error rate of 7.93%, actually worse than the 110-layer network's 6.61%.\n\nResidual connections let networks go deeper, but at a thousand layers, a new problem showed up.\n\nIn March 2016, the original team, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, released this paper, later published at ECCV 2016. This time they went back and wrote out the propagation formulas of the residual unit, used them to work out how the signal could flow most smoothly, and redesigned the residual unit accordingly. After the change, a 1001-layer ResNet reached 4.62% error on CIFAR-10.\n\nThis new version later came to be commonly called ResNet v2, or the pre-activation ResNet.\n\n\n> [!TIP]\n> \n> For ResNet's original design, you can start with our note:\n> \n> - **[[15.12] ResNet: Convolutional with 100 Layers](/blog/papers/classic-cnns/resnet)**\n\n\n## Defining the Problem\n\nFirst, write the original residual unit in general form. The input to the $l$-th unit is $x_l$:\n\n$$\ny_l = h(x_l) + \\mathcal{F}(x_l, \\mathcal{W}_l), \\qquad x_{l+1} = f(y_l)\n$$\n\n$\\mathcal{F}$ is the residual function, for example two $3\\times3$ convolutions; $h$ is the operation on the shortcut; $f$ is the operation after the addition. In the original ResNet, $h(x_l) = x_l$ is an identity mapping and $f$ is ReLU.\n\n[![On the left, diagrams of two residual units. (a) original: x_l runs straight down, and the branch on the right is weight, BN, ReLU, weight, BN in order; after being added to the shortcut, the result goes through ReLU to give x_l+1. (b) proposed: the branch on the right is BN, ReLU, weight, BN, ReLU, weight in order, and the addition gives x_l+1 directly. In both, the shortcut is drawn as a thick gray arrow. On the right are the training curves of a 1001-layer ResNet on CIFAR-10: yellow is original (error 7.61%) and blue is proposed (error 4.92%); dashed lines are training loss and solid lines are test error, and the blue training loss falls much faster right from the start](/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png)](/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png)\n\n*Figure 1 from the paper, excerpted from [**page 2 of the v3 PDF**](https://arxiv.org/pdf/1603.05027v3#page=2). Left: (a) the original residual unit, (b) the residual unit proposed in the paper; the gray arrows are the paths along which information propagates most easily. Right: training curves of a 1001-layer ResNet on CIFAR-10; dashed lines are training loss (left axis), and solid lines are test error (right axis).*\n\nThe curves on the right put the problem in plain view: at the same 1001 layers, the original design's loss falls very slowly early in training and ends at a test error of 7.61%; the modified design's loss falls quickly from the start and ends at 4.92%.\n\nThe two units on the left use exactly the same components; only the order differs. In the original, ReLU comes after the addition, sitting on the shortcut path; the new design moves both BN and ReLU in front of the weight layers and does nothing after the addition. As a result, the gray shortcut passes through no operation at all, from start to finish.\n\nWhy does this matter? The authors' argument starts by treating both $h$ and $f$ as identity mappings.\n\n## Solving the Problem\n\n### Two Identity Mappings\n\nIf $f$ is also an identity mapping, $x_{l+1} = y_l$, and the equation above becomes:\n\n$$\nx_{l+1} = x_l + \\mathcal{F}(x_l, \\mathcal{W}_l)\n$$\n\nUnrolling all the way to a deeper unit $L$:\n\n$$\nx_L = x_l + \\sum_{i=l}^{L-1} \\mathcal{F}(x_i, \\mathcal{W}_i)\n$$\n\nWhat this says: any deep feature $x_L$ equals some shallower feature $x_l$ plus the outputs of all the residual functions in between. Compare an ordinary network, where $x_L$ is the result of a chain of matrix multiplications, roughly $\\prod_i W_i\\, x_0$ (ignoring BN and ReLU). One adds; the other multiplies.\n\nThe same goes for backpropagation. With loss $\\mathcal{E}$, by the chain rule:\n\n$$\n\\frac{\\partial \\mathcal{E}}{\\partial x_l}\n= \\frac{\\partial \\mathcal{E}}{\\partial x_L}\\frac{\\partial x_L}{\\partial x_l}\n= \\frac{\\partial \\mathcal{E}}{\\partial x_L}\\left(1 + \\frac{\\partial}{\\partial x_l}\\sum_{i=l}^{L-1}\\mathcal{F}(x_i, \\mathcal{W}_i)\\right)\n$$\n\nThe 1 in the parentheses delivers $\\partial\\mathcal{E}/\\partial x_L$ straight to the shallower $x_l$ without passing through any weight layer. The other term does pass through weight layers, but it's unlikely to be exactly $-1$ for every sample in a mini-batch and cancel the whole parenthesis out. So even when the weights are very small, the gradient won't vanish.\n\nBoth of these nice properties rest on two conditions: the shortcut $h$ is an identity mapping, and the after-addition $f$ is an identity mapping too. In terms of the figure, the gray arrow must carry no operation other than the addition.\n\nThe authors then check this in two parts: first they change the shortcut $h$, then the after-addition $f$.\n\n### What If the Shortcut Carries Something Extra?\n\nThe simplest way to break it is to multiply the shortcut by a constant $\\lambda_l$: $h(x_l) = \\lambda_l x_l$. Unrolling the same way, the backpropagation equation becomes:\n\n$$\n\\frac{\\partial \\mathcal{E}}{\\partial x_l}\n= \\frac{\\partial \\mathcal{E}}{\\partial x_L}\\left(\\prod_{i=l}^{L-1}\\lambda_i + \\frac{\\partial}{\\partial x_l}\\sum_{i=l}^{L-1}\\hat{\\mathcal{F}}(x_i, \\mathcal{W}_i)\\right)\n$$\n\n$\\hat{\\mathcal{F}}$ absorbs those constants. The original 1 has turned into a product of $\\lambda$'s. In a very deep network, if every $\\lambda$ is greater than 1, this term explodes exponentially; if every $\\lambda$ is less than 1, it vanishes exponentially, and the gradient has to go through the weight layers instead.\n\nResNet-110 has 54 residual units. Let's make the residual functions very small so we see only the effect of the shortcut:\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nD, units = 64, 54                                       # ResNet-110 has 54 residual units\nWs = [torch.randn(D, D) * 0.01 for _ in range(units)]   # deliberately keep the residual functions tiny\n\nfor lam in (1.0, 0.9, 0.5, 1.1):\n    x0 = torch.randn(D, requires_grad=True)\n    x = x0\n    for W in Ws:\n        x = lam * x + torch.relu(x) @ W                 # the shortcut is multiplied by λ\n    x.sum().backward()\n    print(f\"λ = {lam}: gradient reaching the first layer ≈ {x0.grad.abs().mean().item():.3g}, λ^54 = {lam ** units:.3g}\")\n```\n\n```text\nλ = 1.0: gradient reaching the first layer ≈ 0.954, λ^54 = 1\nλ = 0.9: gradient reaching the first layer ≈ 0.00322, λ^54 = 0.00338\nλ = 0.5: gradient reaching the first layer ≈ 5.67e-17, λ^54 = 5.55e-17\nλ = 1.1: gradient reaching the first layer ≈ 162, λ^54 = 172\n```\n\nThe gradient is almost exactly $\\lambda^{54}$. $\\lambda = 0.9$ looks like a mere 10% discount, but after passing through 54 units, only three thousandths are left; $\\lambda = 0.5$ shrinks it to $10^{-17}$. With more complex shortcuts such as gates or $1\\times1$ convolutions, what gets multiplied together becomes each shortcut's derivative, which can block the signal just the same.\n\nThe authors then compare several kinds of shortcut in practice, on ResNet-110 for CIFAR-10:\n\n[![Diagrams of six residual units, each with a branch on the right of 3x3 conv, ReLU, 3x3 conv; they differ in the shortcut: (a) original has no operation; (b) constant scaling multiplies both the shortcut and the branch by 0.5; (c) exclusive gating computes a gate with a 1x1 conv and a sigmoid, multiplying the branch by the gate value and the shortcut by 1 minus the gate value; (d) shortcut-only gating multiplies only the shortcut by 1 minus the gate value; (e) conv shortcut has a 1x1 conv on the shortcut; (f) dropout shortcut has dropout on the shortcut. Every unit is followed by ReLU after the addition](/content/papers/assets/7710a267ec379977c4b41b07fbb2532dd44111584b6c0542b7ce9e96556e4f13.png)](/content/papers/assets/7710a267ec379977c4b41b07fbb2532dd44111584b6c0542b7ce9e96556e4f13.png)\n\n*Figure 2 from the paper, excerpted from [**page 5 of the v3 PDF**](https://arxiv.org/pdf/1603.05027v3#page=5). The kinds of shortcut used in Table 1. For simplicity, the BN after each weight layer isn't drawn.*\n\n| Shortcut | Coefficient on the shortcut | Coefficient on the branch $\\mathcal{F}$ | Error |\n| --- | --- | --- | ---: |\n| (a) Original identity shortcut | 1 | 1 | **6.61%** |\n| (b) Constant scaling | 0 | 1 | fail (a plain network) |\n| (b) Constant scaling | 0.5 | 1 | fail |\n| (b) Constant scaling | 0.5 | 0.5 | 12.35% |\n| (c) Exclusive gating, $b_g$ initialized to $-6$ | $1-g(x)$ | $g(x)$ | 8.70% |\n| (c) Exclusive gating, $b_g$ initialized to 0 through $-5$ | $1-g(x)$ | $g(x)$ | fail |\n| (d) Shortcut-only gating, $b_g$ initialized to 0 | $1-g(x)$ | 1 | 12.86% |\n| (d) Shortcut-only gating, $b_g$ initialized to $-6$ | $1-g(x)$ | 1 | 6.91% |\n| (e) $1\\times1$ convolutional shortcut | $1\\times1$ conv | 1 | 12.22% |\n| (f) Dropout shortcut | dropout 0.5 | 1 | fail |\n\n*Table: Selected rows excerpted from Table 1 of the paper. CIFAR-10 test set, ResNet-110, median of five runs; an error above 20% is marked “fail.” These experiments still use ReLU after the addition.*\n\nA few things stand out:\n\n- **Constant scaling by 0.5**: without scaling the branch, it fails to train; with the branch also multiplied by 0.5 (like a Highway network with its gates frozen), it converges, but the error is 12.35%, nearly twice the baseline.\n- **Exclusive gating** is the Highway Networks approach. The initial value of the gate bias is critical: the authors searched between 0 and $-10$, and even the best value, $-6$, reaches only 8.70%.\n- With **shortcut-only gating**, an initial bias of $-6$ makes $1-g(x)$ close to 1, so the shortcut is nearly an identity mapping, and its 6.91% is also the closest to the baseline.\n- **$1\\times1$ convolutional shortcuts** worked well on the original 34-layer ResNet (option C in the ResNet paper), but stacked over 54 units they give 12.22%. The authors saw something similar with ResNet-101 on ImageNet.\n- A **dropout shortcut** statistically amounts to multiplying the shortcut by an expected value of 0.5, so it blocks the signal just like constant scaling.\n\n\n> [!TIP]\n> \n> For the prototype of exclusive gating, see our note:\n> \n> - **[[15.05] Highway Networks: On the Eve of Residuals](/blog/papers/classic-cnns/highway-networks)**\n\n\nGates and $1\\times1$ convolutions both add parameters, so in theory they should be more expressive; with shortcut-only gating or $1\\times1$ convolutions, the solution space even contains the identity shortcut, which they would only need to learn. Yet their **training** error is higher. So the degradation comes from optimization and has nothing to do with expressiveness.\n\n### Where Should the After-Addition ReLU Go?\n\nIn the previous experiments, the after-addition $f$ was still ReLU, so the derivation held only approximately. Next up is $f$.\n\nThe only way to make $f$ an identity mapping is to rearrange where ReLU and BN go. The authors tried these arrangements:\n\n[![Diagrams of five residual units with the same components in different orders. (a) original: weight, BN, ReLU, weight, BN, then ReLU after the addition. (b) BN after addition: weight, BN, ReLU, weight, then BN and ReLU after the addition. (c) ReLU before addition: weight, BN, ReLU, weight, BN, ReLU, then the addition. (d) ReLU-only pre-activation: ReLU, weight, BN, ReLU, weight, BN, then the addition. (e) full pre-activation: BN, ReLU, weight, BN, ReLU, weight, then the addition](/content/papers/assets/f032506b6dda998779a8abf2fee9cbf44371ac4a2c4ebf8c93d834286bbb7501.png)](/content/papers/assets/f032506b6dda998779a8abf2fee9cbf44371ac4a2c4ebf8c93d834286bbb7501.png)\n\n*Figure 4 from the paper, excerpted from [**page 8 of the v3 PDF**](https://arxiv.org/pdf/1603.05027v3#page=8). The activation placements used in Table 2; all units have the same components, only in different orders.*\n\n| Arrangement | ResNet-110 | ResNet-164 |\n| --- | ---: | ---: |\n| (a) Original residual unit | 6.61% | 5.93% |\n| (b) BN after addition | 8.17% | 6.50% |\n| (c) ReLU before addition | 7.84% | 6.14% |\n| (d) ReLU-only pre-activation | 6.71% | 5.91% |\n| (e) Full pre-activation | **6.37%** | **5.46%** |\n\n*Table: Excerpted from Table 2 of the paper. CIFAR-10 test error. ResNet-164 uses the bottleneck structure ($1\\times1$, $3\\times3$, $1\\times1$), with computation similar to a unit of two $3\\times3$ layers.*\n\n**(b) BN after addition** is an experiment in the opposite direction: $f$ now includes both BN and ReLU. The result is clearly worse. BN now alters the signal on the shortcut, and the loss has a hard time coming down early in training.\n\n**(c) ReLU before addition** is the most direct way to make $f$ an identity mapping, and it does worse too. The problem is that with ReLU at the end of the branch, the residual function's output can never be negative. Intuitively, a “residual” should be able to go either way; if it can only be positive, the signal keeps increasing monotonically. Let's see what the two arrangements look like after stacking ten units:\n\n```python\nimport torch\nimport torch.nn as nn\n\ntorch.manual_seed(0)\nc = 16\n\ndef relu_before_add():          # Fig. 4(c): ReLU moved before the addition\n    return nn.Sequential(nn.Conv2d(c, c, 3, padding=1), nn.BatchNorm2d(c), nn.ReLU(),\n                         nn.Conv2d(c, c, 3, padding=1), nn.BatchNorm2d(c), nn.ReLU())\n\ndef full_preact():              # Fig. 4(e): BN and ReLU both placed before the weight layers\n    return nn.Sequential(nn.BatchNorm2d(c), nn.ReLU(), nn.Conv2d(c, c, 3, padding=1),\n                         nn.BatchNorm2d(c), nn.ReLU(), nn.Conv2d(c, c, 3, padding=1))\n\nx0 = torch.randn(8, c, 16, 16)\nfor name, make in [(\"ReLU before addition\", relu_before_add), (\"Full pre-activation\", full_preact)]:\n    x, lows, means = x0, [], []\n    for unit in range(10):\n        r = make()(x)\n        lows.append(r.min().item())\n        x = x + r\n        means.append(x.mean().item())\n    print(f\"{name}: residual minimum {min(lows):.2f}; output mean {means[0]:.2f} → {means[4]:.2f} → {means[9]:.2f}\")\n```\n\n```text\nReLU before addition: residual minimum 0.00; output mean 0.38 → 1.97 → 3.96\nFull pre-activation: residual minimum -2.33; output mean -0.01 → 0.07 → 0.09\n```\n\nWith ReLU before the addition, the residual bottoms out at exactly 0, and each added unit raises the mean another notch; with pre-activation, the residual takes both signs, and the mean stays near 0.\n\n### A Different Angle: Pre-Activation\n\nThe remaining two arrangements come from a shift in perspective.\n\nIn the original design, the after-addition activation $f$ affects both paths of the next unit:\n\n$$\ny_{l+1} = f(y_l) + \\mathcal{F}(f(y_l), \\mathcal{W}_{l+1})\n$$\n\nIf this activation acts only on the residual branch, with the shortcut bypassing it, we get:\n\n$$\ny_{l+1} = y_l + \\mathcal{F}(\\hat{f}(y_l), \\mathcal{W}_{l+1})\n$$\n\nChanging notation, this is $x_{l+1} = x_l + \\mathcal{F}(\\hat{f}(x_l), \\mathcal{W}_l)$. Nothing comes after the addition, so $f$ has become an identity mapping; what used to be “the previous unit's output activation” has become “the next unit's input activation.”\n\n[![Three columns of diagrams. (a) The original residual unit has an act. after the addition, and this act. affects both the shortcut and the branch of the next unit. (b) The after-addition act. is placed only on the branch leading to the weights (outlined in red), and the shortcut bypasses it. (c) After redrawing the unit boundaries, this is equivalent to a pre-activation residual unit in which each unit applies act. before weight, with no operation after the addition](/content/papers/assets/1a7a520a803ed65de475f2ee4bc5ade9b1833de9aefd758c5ef57153e5d087d2.png)](/content/papers/assets/1a7a520a803ed65de475f2ee4bc5ade9b1833de9aefd758c5ef57153e5d087d2.png)\n\n*Figure 5 from the paper, excerpted from [**page 10 of the v3 PDF**](https://arxiv.org/pdf/1603.05027v3#page=10). Applying the after-addition activation only to the weight branch is equivalent to building a pre-activation residual unit.*\n\nGoing from (b) to (c) in the figure just redraws the unit boundaries. The authors also point out that this before/after distinction only means something in branched structures with additions; an ordinary $N$-layer network has $N-1$ activations, and it makes no difference whether you call them pre-activations or post-activations.\n\nFollowing this idea, (d) in Figure 4 moves only ReLU to the front, with results about the same as the original design; the authors suspect that this ReLU, not paired with BN, doesn't get BN's benefits. (e) moves both BN and ReLU in front of the weight layers, and the results clearly improve. This is the unit the paper proposes.\n\n## Discussion\n\n### The Deeper, the Wider the Gap\n\n| Dataset | Network | Original unit | Pre-activation |\n| --- | --- | ---: | ---: |\n| CIFAR-10 | ResNet-110 (one layer per unit) | 9.90% | 8.91% |\n| CIFAR-10 | ResNet-110 | 6.61% | 6.37% |\n| CIFAR-10 | ResNet-164 | 5.93% | 5.46% |\n| CIFAR-10 | ResNet-1001 | 7.61% | **4.92%** |\n| CIFAR-100 | ResNet-164 | 25.16% | 24.33% |\n| CIFAR-100 | ResNet-1001 | 27.82% | **22.71%** |\n\n*Table: Excerpted from Table 3 of the paper. ResNet-1001 uses the bottleneck structure, with 333 residual units in all, 111 for each feature map size.*\n\nPre-activation is better in every case, and the deeper the network, the bigger the gap: at 1001 layers, CIFAR-10 error falls from 7.61% to 4.92%, and CIFAR-100 error from 27.82% to 22.71%.\n\nThe authors attribute the gains to two things.\n\n**First, easier optimization.** This is most visible at 1001 layers (Figure 1, right). When $f$ is ReLU, negative signals get truncated; with many units, the earlier formulas stop being a good approximation. The 164-layer network is affected much less: its curve suffers a little early in training and quickly recovers. Monitoring the network's responses, the authors found that after some training the weights adjust so that $y_l$ is mostly above 0, and ReLU rarely truncates anything; $x_l$ is already non-negative after the previous ReLU, so $y_l$ drops below 0 only when $\\mathcal{F}$ is strongly negative. At a thousand layers, truncation happens far more often.\n\n**Second, less overfitting.** The pre-activation version converges to a slightly higher training loss but a lower test error. The authors suspect this is BN's regularizing effect: in the original unit, the BN-normalized signal is immediately added to the shortcut, so what actually enters the next weight layer is unnormalized; in the pre-activation version, the input to every weight layer goes through BN first.\n\n### Comparison with Other Methods\n\nOn CIFAR, the authors' ResNet-1001 reaches 4.92% on CIFAR-10 (median of five runs), and 4.62% with the mini-batch size changed to 64; on CIFAR-100 it reaches 22.71%. They stress that these results come without specially tailoring the width or filter sizes and without regularization such as dropout, which is very effective on small datasets; all they rely on is “going deeper.”\n\nOn ImageNet, the authors first note an evaluation detail: the original ResNet was trained with the shorter side scaled to between 256 and 480, yet tested on $224\\times224$ crops from images with a shorter side of 256, which works against ResNet. So this time they take a single $320\\times320$ crop from images with a shorter side of 320. ResNet is fully convolutional and can take larger inputs directly, and this size is also close to Inception-v3's 299.\n\n| Method | Augmentation | Top-1 error | Top-5 error |\n| --- | --- | ---: | ---: |\n| ResNet-152, original unit | scale | 21.3% | 5.5% |\n| ResNet-152, pre-activation | scale | 21.1% | 5.5% |\n| ResNet-200, original unit | scale | 21.8% | 6.0% |\n| ResNet-200, pre-activation | scale | 20.7% | 5.3% |\n| ResNet-200, pre-activation | scale + aspect ratio | **20.1%** | **4.8%** |\n| Inception-v3 | scale + aspect ratio | 21.2% | 5.6% |\n\n*Table: Excerpted from Table 5 of the paper. ILSVRC 2012 validation set, single crop; ResNets are trained at $224\\times224$ and tested at $320\\times320$, while Inception-v3 is trained and tested at $299\\times299$.*\n\nSwitching ResNet-152 to pre-activation improves it only from 21.3% to 21.1%, because it had no obvious generalization problem to begin with. ResNet-200 is a different story: with the original unit, its error is 21.8%, worse than ResNet-152, yet its training error is lower than ResNet-152's, which means it's overfitting. With pre-activation it gets 20.7%, better than both versions of ResNet-152.\n\nThe authors also finished a “BN after addition” version of ResNet-101 on ImageNet: its single-crop error is 24.6%/7.5%, worse than the original's 23.6%/7.1%, consistent with the CIFAR results.\n\n\n> [!TIP]\n> \n> In the paper, the authors also mention the concurrent Inception-ResNet-v2 and expect these findings to help it as well:\n> \n> - **[[16.02] Inception-v4: Much Faster, a Little More Accurate](/blog/papers/classic-cnns/inception-v4)**\n\n\nAs for computational cost, model complexity grows linearly with depth, so 1001 layers cost about ten times as much as 100 layers. ResNet-1001 on CIFAR takes about 27 hours to train on two GPUs; ResNet-200 on ImageNet takes about three weeks on eight GPUs.\n\n## Conclusion\n\nThis paper's conclusion fits in one sentence: keep the shortcut clean. Don't multiply the shortcut by constants, don't add gates, don't put convolutions or dropout on it; and don't follow the addition with ReLU or BN either. That way, in both forward and backward propagation, the signal can travel directly from any unit to any other.\n\nIn practice, the change is small: move BN and ReLU forward in the residual branch, and do nothing after the addition. If you're training a very deep ResNet, or your model has started to overfit, try the pre-activation version first; for shallower networks, the two make little difference.\n\nThe principle of “put nothing on the shortcut” came up again a few years later with LayerNorm in Transformers; if you're interested, continue with the [Pre-LN](/blog/papers/transformers/pre-ln) note.\n","imageSizes":{"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png":{"width":967,"height":469},"/content/papers/assets/7710a267ec379977c4b41b07fbb2532dd44111584b6c0542b7ce9e96556e4f13.png":{"width":690,"height":659},"/content/papers/assets/f032506b6dda998779a8abf2fee9cbf44371ac4a2c4ebf8c93d834286bbb7501.png":{"width":965,"height":466},"/content/papers/assets/1a7a520a803ed65de475f2ee4bc5ade9b1833de9aefd758c5ef57153e5d087d2.png":{"width":967,"height":546}}}