{"notes":[{"id":"classic-cnns/inception-v4","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v4","subtitle":"Much Faster, a Little More Accurate","author":"Z. Yuan","paperDate":"2016-02-23","publishedOn":"2026-01-27","world":"external/inception-v4","paperTitle":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","paperUrl":"https://arxiv.org/abs/1602.07261","bibliography":{"title":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","authors":["Christian Szegedy","Sergey Ioffe","Vincent Vanhoucke","Alex Alemi"],"url":"https://arxiv.org/abs/1602.07261","pdfUrl":"https://arxiv.org/pdf/1602.07261v2","codeUrl":null,"preprint":{"id":"1602.07261","submittedOn":"2016-02-23","url":"https://arxiv.org/abs/1602.07261"},"publication":null,"citation":"@misc{classiccnnsinceptionv42016,\n  title = {{Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning}},\n  author = {Christian Szegedy and Sergey Ioffe and Vincent Vanhoucke and Alex Alemi},\n  year = {2016},\n  eprint = {1602.07261},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1602.07261}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v4.ed8e0294ae0b.bib"},"thumbnail":{"src":"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png","width":666,"height":656},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v4.f625ed245ee7.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"dropping-the-baggage-inception-v4","text":"Dropping the Baggage: Inception-v4","depth":3},{"id":"the-entrance","text":"The Entrance","depth":3},{"id":"wiring-residuals-into-inception","text":"Wiring Residuals into Inception","depth":3},{"id":"scaling-down-the-residuals-to-steady-training","text":"Scaling Down the Residuals to Steady Training","depth":3},{"id":"training-setup","text":"Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"residuals-make-training-much-faster","text":"Residuals Make Training Much Faster","depth":3},{"id":"single-model-results","text":"Single-Model Results","depth":3},{"id":"ensembles","text":"Ensembles","depth":3},{"id":"are-residual-connections-necessary","text":"Are Residual Connections “Necessary”?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"Convolutional with 100 Layers","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"en","path":"/content/papers/en/classic-cnns/resnet.b65375cb9846.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-is-residual","text":"What is Residual?","depth":3},{"id":"residual-learning","text":"Residual Learning","depth":3},{"id":"residual-module","text":"Residual Module","depth":3},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"architectures-and-computation","text":"Architectures and Computation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"deeper-bottleneck-architectures","text":"Deeper Bottleneck Architectures","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"Training deeper convolutional networks"},{"id":"classic-cnns/densenet","relation":"next","topic":"Training deeper convolutional networks"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"Big Kernels Bow Out","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2025-12-16","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":null,"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v3.6327335083ff.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"splitting-55-into-two-33-layers","text":"Splitting 5×5 into Two 3×3 Layers","depth":3},{"id":"splitting-further-into-1n-and-n1","text":"Splitting Further into 1×n and n×1","depth":3},{"id":"what-auxiliary-classifiers-really-do","text":"What Auxiliary Classifiers Really Do","depth":3},{"id":"shrinking-the-feature-map-without-choking-the-information","text":"Shrinking the Feature Map Without Choking the Information","depth":3},{"id":"putting-the-pieces-together","text":"Putting the Pieces Together","depth":3},{"id":"keeping-the-model-from-getting-too-sure-of-itself","text":"Keeping the Model from Getting Too Sure of Itself","depth":3},{"id":"training-setup","text":"Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"adding-one-change-at-a-time","text":"Adding One Change at a Time","depth":3},{"id":"how-much-does-lower-resolution-cost","text":"How Much Does Lower Resolution Cost?","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"Keep the Shortcut Clean","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"en","path":"/content/papers/en/classic-cnns/resnet-v2.1b56c20d8b4c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"two-identity-mappings","text":"Two Identity Mappings","depth":3},{"id":"what-if-the-shortcut-carries-something-extra","text":"What If the Shortcut Carries Something Extra?","depth":3},{"id":"where-should-the-after-addition-relu-go","text":"Where Should the After-Addition ReLU Go?","depth":3},{"id":"a-different-angle-pre-activation","text":"A Different Angle: Pre-Activation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-deeper-the-wider-the-gap","text":"The Deeper, the Wider the Gap","depth":3},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]}],"text":"The ILSVRC 2015 classification winner was Microsoft Research's ResNet. Its architecture is quite traditional, convolutions stacked layer upon layer, with residual connections as the only addition; as a single model, it scores about the same as the Inception-v3 that Google had just published.\n\nThe two lines of work each have their own approach: Inception keeps computation down with multi-branch modules and carefully budgeted channel counts; ResNet uses residual connections to make very deep networks trainable.\n\nSo what happens if you combine them?\n\nIn February 2016, Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi of Google released this paper, later published at AAAI 2017. The title spells out all three topics: the new Inception-v4, Inception-ResNet with residual connections added, and what effect residual connections actually have on training.\n\n\n> [!TIP]\n> \n> For this paper's two predecessors, you can start with our notes:\n> \n> - **[[15.12] ResNet: Convolutional with 100 Layers](/blog/papers/classic-cnns/resnet)**\n> - **[[15.12] Inception-v3: Big Kernels Bow Out](/blog/papers/classic-cnns/inception-v3)**\n\n\n## Defining the Problem\n\nThis paper tackles two problems at once.\n\nThe first is residual connections. The ResNet authors argue that residual connections are **inherently necessary** for training very deep convolutional networks. Inception networks are very deep to begin with, so replacing the “concatenate the channels” step at the end of an Inception module with “add back the input” seems natural. The question is: what does it buy you? Is the benefit in final accuracy, or somewhere else?\n\nThe second is Inception itself. The authors admit that Inception-v3 carries a lot of historical baggage. Earlier models were trained with DistBelief, where each replica had to be split into several sub-networks to fit in memory, and the number of filters in each layer was carefully tuned to balance computation across the sub-networks. After the move to TensorFlow, that constraint was gone. On top of that, they had always been conservative about changing the architecture, modifying one component at a time while keeping the rest fixed, and the network grew more and more complicated as a result.\n\nNow that the baggage can be set down, can Inception become simpler, deeper, and wider?\n\n## Solving the Problem\n\n### Dropping the Baggage: Inception-v4\n\n[![The overall Inception-v4 architecture, from bottom to top: Input 299x299x3, Stem (output 35x35x384), 4 x Inception-A (35x35x384), Reduction-A (17x17x1024), 7 x Inception-B (17x17x1024), Reduction-B (8x8x1536), 3 x Inception-C (8x8x1536), Average Pooling (1536), Dropout (keep 0.8), Softmax (1000)](/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png)](/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png)\n\n*Figure 9 from the paper, excerpted from [**page 5 of the v2 PDF**](https://arxiv.org/pdf/1602.07261v2#page=5). The overall Inception-v4 architecture, with the output size of each stage on the right.*\n\nThis looks a lot like Inception-v3's architecture table: three resolutions ($35\\times35$, $17\\times17$, $8\\times8$), several Inception modules stacked at each, and reduction modules in between to move to the next resolution. The difference is that Inception-v4 **uses just one kind of module per resolution**, and more of them: 4 Inception-A, 7 Inception-B, and 3 Inception-C.\n\nThe three modules carry on Inception-v3's factorization methods:\n\n- **Inception-A** ($35\\times35$): four branches, the longest of which replaces $5\\times5$ with two $3\\times3$ layers; each branch outputs 96 channels, for 384 in total.\n- **Inception-B** ($17\\times17$): uses asymmetric $1\\times7$ and $7\\times1$ convolutions, for 1024 channels in total.\n- **Inception-C** ($8\\times8$): outputs $1\\times3$ and $3\\times1$ side by side, for 1536 channels in total.\n\nConvolutions marked “V” in the figures use valid padding: no zero padding, so the output shrinks. Unmarked ones use same padding, so the output is the same size as the input.\n\n### The Entrance\n\nCompared with Inception-v3, the most visible change is at the start of the network. The authors call it the stem:\n\n[![The Inception-v4 stem, from bottom to top: Input 299x299x3; 3x3 Conv (32 stride 2 V) gives 149x149x32; 3x3 Conv (32 V) gives 147x147x32; 3x3 Conv (64) gives 147x147x64; it then splits into two paths, 3x3 MaxPool (stride 2 V) and 3x3 Conv (96 stride 2 V), and Filter concat gives 73x73x160; it splits again into two paths, 1x1 Conv (64) followed by 3x3 Conv (96 V), and 1x1 Conv (64), 7x1 Conv (64), 1x7 Conv (64), 3x3 Conv (96 V), and Filter concat gives 71x71x192; finally it splits into 3x3 Conv (192 V) and MaxPool (stride=2 V), and Filter concat gives 35x35x384](/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png)](/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png)\n\n*Figure 3 from the paper, excerpted from [**page 3 of the v2 PDF**](https://arxiv.org/pdf/1602.07261v2#page=3). The stem shared by Inception-v4 and Inception-ResNet-v2.*\n\nThe first three layers are the same as in Inception-v3, going from $299\\times299$ to $147\\times147\\times64$. After that come three rounds in a row of “split, then concatenate”: each time, either one branch takes care of the shrinking, or two branches each process the input before merging. Let's walk through the figure:\n\n```python\nimport torch\nimport torch.nn as nn\n\ndef conv(cin, cout, k, stride=1, valid=False):\n    # Convolutions marked V use no padding; the others are padded so the output size matches the input\n    pad = 0 if valid else (k[0] // 2, k[1] // 2) if isinstance(k, tuple) else k // 2\n    return nn.Sequential(nn.Conv2d(cin, cout, k, stride, pad), nn.ReLU())\n\nx = torch.randn(1, 3, 299, 299)\n\nx = conv(3, 32, 3, stride=2, valid=True)(x)\nx = conv(32, 32, 3, valid=True)(x)\nx = conv(32, 64, 3)(x)\nprint(\"After three convolutions:\", tuple(x.shape[1:]))\n\n# First side-by-side split: max pooling | stride-2 convolution\na = nn.MaxPool2d(3, 2)(x)\nb = conv(64, 96, 3, stride=2, valid=True)(x)\nx = torch.cat([a, b], dim=1)\nprint(\"Pool\", tuple(a.shape[1:]), \"+ conv\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n\n# Second side-by-side split: two convolution branches\na = nn.Sequential(conv(160, 64, 1), conv(64, 96, 3, valid=True))(x)\nb = nn.Sequential(conv(160, 64, 1), conv(64, 64, (7, 1)), conv(64, 64, (1, 7)), conv(64, 96, 3, valid=True))(x)\nx = torch.cat([a, b], dim=1)\nprint(\"Branch 1\", tuple(a.shape[1:]), \"+ branch 2\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n\n# Third side-by-side split: stride-2 convolution | max pooling\na = conv(192, 192, 3, stride=2, valid=True)(x)    # stride must be 2 to match the pooling path's 35×35\nb = nn.MaxPool2d(3, 2)(x)\nx = torch.cat([a, b], dim=1)\nprint(\"Conv\", tuple(a.shape[1:]), \"+ pool\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n```\n\n```text\nAfter three convolutions: (64, 147, 147)\nPool (64, 73, 73) + conv (96, 73, 73) → (160, 73, 73)\nBranch 1 (96, 71, 71) + branch 2 (96, 71, 71) → (192, 71, 71)\nConv (192, 35, 35) + pool (192, 35, 35) → (384, 35, 35)\n```\n\nThe output shapes match the $73\\times73\\times160$, $71\\times71\\times192$, and $35\\times35\\times384$ marked in the figure. Pooling doesn't change the channel count, so the first merge gives $64+96=160$ and the last gives $192+192=384$.\n\nThe first and third merges use Inception-v3's method for shrinking feature maps: a stride-2 convolution and a stride-2 pooling side by side, avoiding the information bottleneck that pooling first would create. This time even the start of the network does it.\n\n### Wiring Residuals into Inception\n\nFor the residual versions of Inception, the authors use cheaper modules. Take Inception-ResNet-v2's $35\\times35$ module as an example:\n\n[![The Inception-ResNet-A module: the Relu activation at the bottom splits into four paths. The leftmost goes straight to the plus sign at the top; the other three branches are 1x1 Conv (32); 1x1 Conv (32) followed by 3x3 Conv (32); and 1x1 Conv (32) followed by 3x3 Conv (48) and then 3x3 Conv (64). The outputs of the three branches go together into a 1x1 Conv (384 Linear), are added to the pass-through path on the far left, and finally pass through a Relu activation](/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png)](/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png)\n\n*Figure 16 from the paper, excerpted from [**page 8 of the v2 PDF**](https://arxiv.org/pdf/1602.07261v2#page=8). The $35\\times35$ module of Inception-ResNet-v2 (Inception-ResNet-A).*\n\nCompared with a pure Inception module, two things are different:\n\n1. The branches are narrower. The three branches end up with only $32+32+64=128$ channels, far fewer than the input's 384.\n2. After the branches are concatenated, there's an extra $1\\times1$ convolution **with no activation function** that expands the channels back to 384 so the result can be added to the input. The authors call it a filter-expansion layer; it makes up for the dimensionality reduction caused by the Inception module.\n\n```python\nimport torch\nimport torch.nn as nn\n\ndef conv(cin, cout, k):\n    return nn.Sequential(nn.Conv2d(cin, cout, k, padding=k // 2, bias=False), nn.BatchNorm2d(cout), nn.ReLU())\n\nclass InceptionResNetA(nn.Module):\n    def __init__(self, channels=384, scale=0.1):\n        super().__init__()\n        self.b1 = conv(channels, 32, 1)\n        self.b2 = nn.Sequential(conv(channels, 32, 1), conv(32, 32, 3))\n        self.b3 = nn.Sequential(conv(channels, 32, 1), conv(32, 48, 3), conv(48, 64, 3))\n        self.expand = nn.Conv2d(32 + 32 + 64, channels, 1)   # 1×1, no activation function\n        self.scale = scale\n\n    def forward(self, x):\n        branches = torch.cat([self.b1(x), self.b2(x), self.b3(x)], dim=1)\n        residual = self.expand(branches)\n        print(\"Concatenated branches:\", tuple(branches.shape[1:]), \"→ expanded by 1×1:\", tuple(residual.shape[1:]))\n        return torch.relu(x + self.scale * residual)\n\nx = torch.randn(2, 384, 35, 35)\nprint(\"Output:\", tuple(InceptionResNetA()(x).shape[1:]))\n```\n\n```text\nConcatenated branches: (128, 35, 35) → expanded by 1×1: (384, 35, 35)\nOutput: (384, 35, 35)\n```\n\nThe `scale` in the code is the star of the next section; set it aside for now.\n\nThe authors tried several residual versions, but the paper details only two:\n\n| Network | Comparable cost | Stem | Body |\n| --- | --- | --- | --- |\n| Inception-ResNet-v1 | About the same as Inception-v3 | Its own stem (Figure 14) | 5 × A, Reduction-A, 10 × B, Reduction-B, 5 × C |\n| Inception-ResNet-v2 | About the same as Inception-v4 | Same as Inception-v4 (Figure 3) | As above, with wider modules |\n\n*Table: Compiled from Section 3.2 and Figure 15 of the paper. The two networks share the same overall structure; their modules differ in channel counts.*\n\nAlthough Inception-ResNet-v2 and Inception-v4 have comparable raw computational cost, the authors note that in practice Inception-v4 is much slower per training step, probably because it has more layers.\n\nOne more detail: in Inception-ResNet, batch normalization is applied only on top of the ordinary layers, **not after the additions**. The authors say that using batch normalization throughout should be better, but on layers with large activations it takes up a disproportionate amount of GPU memory. To keep each model replica on a single GPU, they dropped those BN layers, which let them stack many more modules.\n\n### Scaling Down the Residuals to Steady Training\n\nNext, the authors ran into trouble: when the number of filters exceeded 1000, the residual versions became unstable, and the network would “die” early in training. Concretely, after a few tens of thousands of iterations, the last layer before average pooling output nothing but zeros. Neither lowering the learning rate nor adding an extra batch normalization to that layer could bring it back.\n\nWhat worked was simple: multiply the residual by a small constant before adding it.\n\n[![Diagram of residual scaling: the Relu activation at the bottom splits into two paths; one goes straight to the plus sign at the top, while the other passes through an Inception block and then an Activation Scaling block before reaching the plus sign; the addition is followed by a Relu activation](/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png)](/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png)\n\n*Figure 20 from the paper, excerpted from [**page 9 of the v2 PDF**](https://arxiv.org/pdf/1602.07261v2#page=9). Scaling the residual inside an Inception-ResNet module; the constant is usually around 0.1.*\n\nThe authors generally pick values between 0.1 and 0.3. When the ResNet paper ran into instability with very deep networks, it used two-phase training: first a “warm-up” at a very low learning rate, then a higher learning rate. The authors found that with very many filters, even a learning rate of 0.00001 couldn't hold off the instability, and the high learning rate that followed could undo whatever the warm-up achieved. Scaling the residuals was far more reliable by comparison. Even where the network could train without scaling, adding it never hurt final accuracy, and it made training more stable.\n\nThe paper doesn't analyze why the networks “die.” One thing is easy to see, though: Inception-ResNet's branches contain batch normalization, so their outputs have a roughly fixed magnitude, while the main path has no batch normalization after its additions, so it accumulates once per module. Let's use a simplified model to see how the signal on the main path changes:\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nx0 = torch.randn(4096, 256)                     # the signal on the main path: 4096 positions, 256 channels\n\nfor scale in (1.0, 0.3, 0.1):\n    x = x0.clone()\n    stds = []\n    for block in range(1, 41):\n        w = torch.randn(256, 256) / 16\n        r = torch.relu(x) @ w\n        r = (r - r.mean(0)) / r.std(0)          # the branch has BN: each channel's output is pulled back to std 1\n        x = x + scale * r                       # no BN after the addition on the main path\n        if block in (10, 20, 40):\n            stds.append(f\"{x.std().item():5.2f}\")\n    print(f\"Scale {scale}: main-path std after blocks 10, 20, 40:\", \" \".join(stds))\n```\n\n```text\nScale 1.0: main-path std after blocks 10, 20, 40:  3.31  4.57  6.42\nScale 0.3: main-path std after blocks 10, 20, 40:  1.37  1.67  2.15\nScale 0.1: main-path std after blocks 10, 20, 40:  1.05  1.10  1.18\n```\n\nWithout scaling, each module adds a signal about as large as the main path itself, so after 40 modules the standard deviation is about $\\sqrt{41}\\approx6.4$. With a scale of 0.1, each module makes only a small correction, and the signal on the main path still comes mostly down the pass-through path. This is only a simplified illustration at initialization and can't explain the “dying” partway through training; what the paper does establish is the experimental result: with scaling, wide residual networks train stably.\n\n### Training Setup\n\nAlmost the same as Inception-v3: TensorFlow distributed training, this time with 20 replicas, each running on one NVIDIA Kepler GPU. The best models use RMSProp (decay 0.9, $\\epsilon=1.0$) with a learning rate of 0.045, multiplied by 0.94 every two epochs; evaluation uses a running average of the parameters.\n\n## Discussion\n\n### Residuals Make Training Much Faster\n\n[![A line chart with epochs on the horizontal axis (20 to 200) and top-1 error on the vertical axis (15% to 29%). The green dashed line is inception-v3, and the red solid line is inception-resnet-v1. The red line falls much faster, nearing 22% by around 80 epochs, while the green line doesn't catch up until around 160 epochs; both lines end up just above 21%](/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png)](/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png)\n\n*Figure 21 from the paper, excerpted from [**page 9 of the v2 PDF**](https://arxiv.org/pdf/1602.07261v2#page=9). Top-1 error (single crop) during training for Inception-v3 and Inception-ResNet-v1, which have similar computational cost.*\n\nThe red line is Inception-ResNet-v1, with residuals; the green line is Inception-v3. The red line falls much faster, yet both lines end up in about the same place: on top-1 the residual version is slightly worse, while on top-5 it's slightly better. For the Inception-v4 and Inception-ResNet-v2 pair, too, the residual version trains faster and finishes slightly better.\n\nComparing all four models, the authors conclude: the residual versions converge faster, but final accuracy depends mainly on model size.\n\n### Single-Model Results\n\n| Network | Top-1 error | Top-5 error |\n| --- | ---: | ---: |\n| BN-Inception | 25.2% | 7.8% |\n| Inception-v3 | 21.2% | 5.6% |\n| Inception-ResNet-v1 | 21.3% | 5.5% |\n| Inception-v4 | 20.0% | 5.0% |\n| **Inception-ResNet-v2** | **19.9%** | **4.9%** |\n\n*Table: Excerpted from Table 2 of the paper. Single model, single crop, on the validation set excluding blacklisted images.*\n\nLooking at pairs of similar computational cost: Inception-v3 and Inception-ResNet-v1 are nearly tied, and Inception-v4 and Inception-ResNet-v2 are only 0.1 percentage points apart. The gap between the two pairs is much larger than the gap within each pair between the versions with and without residuals.\n\nMulti-crop results follow the same trend:\n\n| Network | Crops | Top-1 error | Top-5 error |\n| --- | ---: | ---: | ---: |\n| ResNet-152 | 10 | 21.4% | 5.7% |\n| Inception-v3 | 12 | 19.8% | 4.6% |\n| Inception-ResNet-v1 | 12 | 19.8% | 4.6% |\n| Inception-v4 | 12 | 18.7% | 4.2% |\n| Inception-ResNet-v2 | 12 | 18.7% | 4.1% |\n| ResNet-152 | dense | 19.4% | 4.5% |\n| Inception-v3 | 144 | 18.9% | 4.3% |\n| Inception-ResNet-v1 | 144 | 18.8% | 4.3% |\n| Inception-v4 | 144 | 17.7% | 3.8% |\n| Inception-ResNet-v2 | 144 | 17.8% | 3.7% |\n\n*Table: Excerpted from Tables 3 and 4 of the paper, on the full 50,000-image validation set.*\n\n### Ensembles\n\n| Network | Models | Top-1 error | Top-5 error |\n| --- | ---: | ---: | ---: |\n| ResNet-152 | 6 | – | 3.6% |\n| Inception-v3 | 4 | 17.3% | 3.6% |\n| **Inception-v4 + 3 × Inception-ResNet-v2** | **4** | **16.5%** | **3.1%** |\n\n*Table: Excerpted from Table 5 of the paper, on the full 50,000-image validation set. ResNet uses dense evaluation; the Inception models use 144 crops.*\n\nOne Inception-v4 plus three Inception-ResNet-v2 gives 3.1% top-5 error on the validation set. This ensemble was also submitted to the test server, scoring 3.08%. The authors make a point of saying that in the past year they had submitted results to the test server only twice, once for the BN-Inception paper and once for the ILSVRC 2015 CLSLOC competition, so the test-set number can be taken as a true estimate of generalization.\n\nThe authors also notice that single-model improvements become less pronounced once models are ensembled.\n\n### Are Residual Connections “Necessary”?\n\nBack to the first question from the beginning. The authors' answer is direct: at least for image recognition, their results don't support the ResNet paper's claim that residual connections are inherently necessary. Inception-v4 has no residual connections, yet it trains just fine and performs on par with Inception-ResNet-v2.\n\nThey also hedge: seeing the full extent of what residual connections offer may take deeper architectures and more measurement points. What is certain is that residual connections make training much faster, and that alone is reason enough to use them.\n\n## Conclusion\n\nThis paper delivers three networks: the pure-Inception Inception-v4, and Inception-ResNet-v1 and v2, whose computational costs match Inception-v3 and Inception-v4, respectively.\n\nFirst, the conditions of the comparison need to be clear: the Inception-ResNet modules were picked “in a somewhat ad hoc manner” (the authors' own words), with the only requirement being a computational cost close to the corresponding pure Inception model; the authors also tried bigger and wider versions, which performed very similarly on ImageNet. Under these conditions, what residual connections mainly bring is training speed. What really opens up a gap is a bigger model: Inception-v4 and Inception-ResNet-v2 are both clearly better than the v3-class models.\n\nThe conclusions are practical for anyone putting these models to use: if training time matters, add residual connections; if final accuracy matters, model size counts for more than whether there are residuals. When the network is very wide, remember to multiply the residual by 0.1 to 0.3 before adding it back, or training may “die” partway through.\n\nAs for whether residuals are a necessity for training deep networks, ResNet and this paper each have their own view. Around the same time, the ResNet authors themselves were studying the right way to wire residuals, which is the subject of the next note, [ResNet v2](/blog/papers/classic-cnns/resnet-v2).\n","imageSizes":{"/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png":{"width":335,"height":941},"/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png":{"width":431,"height":1189},"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png":{"width":666,"height":656},"/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png":{"width":277,"height":396},"/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png":{"width":565,"height":356}}}