{"notes":[{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"Big Kernels Bow Out","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2025-12-16","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":null,"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v3.6327335083ff.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"splitting-55-into-two-33-layers","text":"Splitting 5×5 into Two 3×3 Layers","depth":3},{"id":"splitting-further-into-1n-and-n1","text":"Splitting Further into 1×n and n×1","depth":3},{"id":"what-auxiliary-classifiers-really-do","text":"What Auxiliary Classifiers Really Do","depth":3},{"id":"shrinking-the-feature-map-without-choking-the-information","text":"Shrinking the Feature Map Without Choking the Information","depth":3},{"id":"putting-the-pieces-together","text":"Putting the Pieces Together","depth":3},{"id":"keeping-the-model-from-getting-too-sure-of-itself","text":"Keeping the Model from Getting Too Sure of Itself","depth":3},{"id":"training-setup","text":"Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"adding-one-change-at-a-time","text":"Adding One Change at a Time","depth":3},{"id":"how-much-does-lower-resolution-cost","text":"How Much Does Lower Resolution Cost?","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]},{"id":"classic-cnns/inception-v1","category":"classic-cnns","legacyPath":"classic-cnns/1409-inception-v1","legacySlug":"1409-inception-v1","legacyPaths":[],"title":"GoogLeNet","subtitle":"Also Known as Inception-V1","author":"Z. Yuan","paperDate":"2014-09-17","publishedOn":"2024-09-22","paperTitle":"Going Deeper with Convolutions","paperUrl":"https://arxiv.org/abs/1409.4842","bibliography":{"title":"Going Deeper with Convolutions","authors":["Christian Szegedy","Wei Liu","Yangqing Jia","Pierre Sermanet","Scott Reed","Dragomir Anguelov","Dumitru Erhan","Vincent Vanhoucke","Andrew Rabinovich"],"url":"https://arxiv.org/abs/1409.4842","pdfUrl":"https://arxiv.org/pdf/1409.4842","codeUrl":null,"preprint":{"id":"1409.4842","submittedOn":"2014-09-17","url":"https://arxiv.org/abs/1409.4842"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2015.7298594"},"citation":"@misc{classiccnnsinceptionv12014,\n  title = {{Going Deeper with Convolutions}},\n  author = {Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich},\n  year = {2014},\n  eprint = {1409.4842},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1409.4842}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v1.3d20e0e32bcd.bib"},"thumbnail":{"src":"/content/papers/assets/254b970b1374c24267b60c2d684338760e9ba8b10dad761a5fa15f1bb8bcf6ef.jpg","width":1330,"height":394},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v1.41dca1189319.md","headings":[{"id":"problem-solved","text":"Problem Solved","depth":2},{"id":"the-inception-module","text":"The Inception Module","depth":3},{"id":"configuration-parameters","text":"Configuration Parameters","depth":3},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"performance-at-ilsvrc-2014","text":"Performance at ILSVRC 2014","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnext","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"}]},{"id":"normalization/batchnorm","category":"normalization","legacyPath":"normalization/1502-batchnorm","legacySlug":"1502-batchnorm","legacyPaths":[],"title":"BatchNorm","subtitle":"Batch Normalization","author":"Z. Yuan","paperDate":"2015-02-11","publishedOn":"2024-08-01","paperTitle":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","paperUrl":"https://arxiv.org/abs/1502.03167","bibliography":{"title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","authors":["Sergey Ioffe","Christian Szegedy"],"url":"https://arxiv.org/abs/1502.03167","pdfUrl":"https://arxiv.org/pdf/1502.03167","codeUrl":null,"preprint":{"id":"1502.03167","submittedOn":"2015-02-11","url":"https://arxiv.org/abs/1502.03167"},"publication":null,"citation":"@misc{normalizationbatchnorm2015,\n  title = {{Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift}},\n  author = {Sergey Ioffe and Christian Szegedy},\n  year = {2015},\n  eprint = {1502.03167},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1502.03167}\n}\n","citationPath":"/content/papers/citations/normalization/batchnorm.bfa8abb33f32.bib"},"thumbnail":{"src":"/content/papers/assets/48b236d4b897164b93764eccfe22db29e5b9bce290d864336b8f12ec127c1e4b.jpg","width":1016,"height":824},"locale":"en","path":"/content/papers/en/normalization/batchnorm.149073a22595.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"increasing-learning-rates","text":"Increasing Learning Rates","depth":3},{"id":"no-need-for-dropout","text":"No Need for Dropout","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"mnist-experiment","text":"MNIST Experiment","depth":3},{"id":"imagenet-experiment","text":"ImageNet Experiment","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/layernorm","relation":"next","topic":"Choosing the normalization axes"}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"Making Convolutions Great Again","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"en","path":"/content/papers/en/classic-cnns/convnext.1a8312c7b164.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"redesigning-network-architecture","text":"Redesigning Network Architecture","depth":3},{"id":"introducing-modern-training-techniques","text":"Introducing Modern Training Techniques","depth":3},{"id":"macro-design","text":"Macro Design","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"large-kernel-convolution","text":"Large Kernel Convolution","depth":3},{"id":"micro-design","text":"Micro Design","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"summary-of-model-configurations","text":"Summary of Model Configurations","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"reparameterization/replknet","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]}],"text":"When GoogLeNet won the ILSVRC 2014 classification challenge, it used only about 5 million parameters, a twelfth of AlexNet's.\n\nThat efficiency is tempting, but anyone who tries to modify it soon runs into trouble. The GoogLeNet paper doesn't clearly explain how each design decision was reached, and the Inception module has many branches. The most intuitive way to scale the model up is to double the number of filters in every group, which quadruples both computation and parameters.\n\nIn December 2015, Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, and Jonathon Shlens of Google, together with Zbigniew Wojna of University College London, released this paper, later published at CVPR 2016. The “Rethinking” in the title refers to exactly this: taking Inception apart and thinking it through again, to find which approaches make the extra computation worthwhile as the network grows.\n\n\n> [!TIP]\n> \n> For what the basic Inception module looks like, you can start with our note:\n> \n> - **[[14.09] GoogLeNet: Also Known as Inception-V1](/blog/papers/classic-cnns/inception-v1)**\n\n\n## Defining the Problem\n\nThe question the authors want to answer: with a limited compute budget, how should a network be scaled up?\n\nFrom a large number of architecture experiments, they distilled four principles. The authors themselves are cautious about them: the principles are still partly speculative, more experiments are needed to establish where they apply, and they suggest consulting them only when in doubt. Still, in their experiments, networks that clearly violated them tended to get worse, and got better again once corrected.\n\n1. **Avoid representational bottlenecks, especially early in the network.** From input to output, the size of the representation should shrink gradually instead of being squeezed flat all at once at some layer. The authors also note that dimensionality is only a rough estimate of information content.\n2. **Higher-dimensional representations are easier to process locally.** More activations per position make features easier to disentangle, and training goes faster.\n3. **Spatial aggregation can be done in lower dimensions.** For example, reducing the number of channels before a $3\\times3$ convolution doesn't hurt expressiveness much. The authors' guess: outputs at adjacent positions are highly correlated, so little information is lost in the reduction.\n4. **Balance width and depth.** Increasing both helps; at a fixed computational cost, increasing width and depth together works best.\n\nMuch of GoogLeNet's early gain came from heavy use of $1\\times1$ convolutions for dimensionality reduction, which is principle 3. The authors see dimensionality reduction as a special case of “factorizing convolutions,” and next they go looking for other ways to factorize.\n\n## Solving the Problem\n\n### Splitting 5×5 into Two 3×3 Layers\n\nLarge kernels are expensive. With the same numbers of input and output channels, a $5\\times5$ convolution costs $25/9\\approx2.78$ times as much as a $3\\times3$ one.\n\nBut large kernels also see farther. Can we get the same field of view with fewer parameters?\n\nThe authors look at a $5\\times5$ convolution from another angle: each output is a small fully connected network sliding over a $5\\times5$ region of the input. Since this is a vision network, the small network can itself exploit translation invariance and be split into two layers: the first is a $3\\times3$ convolution, and the second is fully connected over the $3\\times3$ grid of the first layer's output. Slide this two-layer mini-network over the whole input, and you get two $3\\times3$ convolutional layers.\n\n[![Three layers of blue grids stacked together. The bottom layer is a 5×5 input region, the middle is a 3×3 grid, and the top is a single output cell; red lines show the top cell connecting to the whole 3×3 middle region, and each middle cell in turn connecting to a 3×3 region below](/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png)](/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png)\n\n*Figure 1 from the paper, excerpted from [**page 3 of the v3 PDF**](https://arxiv.org/pdf/1512.00567v3#page=3). The mini-network that replaces a $5\\times5$ convolution: one output in the top layer looks at a $3\\times3$ region of the middle layer, and each middle cell in turn looks at a $3\\times3$ region of the input.*\n\nFirst, how much does it save? Take a $35\\times35$ feature map with 64 channels and keep the number of output channels the same:\n\n```python\nimport torch\nimport torch.nn as nn\n\nx = torch.randn(1, 64, 35, 35)                 # a 35×35 feature map with 64 channels\n\nbig = nn.Conv2d(64, 64, 5, padding=2, bias=False)\nsmall = nn.Sequential(nn.Conv2d(64, 64, 3, padding=1, bias=False),\n                      nn.Conv2d(64, 64, 3, padding=1, bias=False))\n\ndef params(m):\n    return sum(p.numel() for p in m.parameters())\n\nprint(big(x).shape, small(x).shape)\nprint(\"Parameters, 5×5:\", params(big))\nprint(\"Parameters, two 3×3:\", params(small), f\"({params(small) / params(big):.0%})\")\n```\n\n```text\ntorch.Size([1, 64, 35, 35]) torch.Size([1, 64, 35, 35])\nParameters, 5×5: 102400\nParameters, two 3×3: 73728 (72%)\n```\n\nThe output shapes match, and the parameters drop to $(9+9)/25 = 72\\%$, a 28% saving. In a convolutional network, each weight is used exactly once at each output position, so computation drops by the same proportion. This assumes the middle layer has as many channels as the input; the paper's analysis makes the same simplification.\n\nNow check the field of view. Set the weights of both $3\\times3$ layers to 1 and look at the gradient of the single center output with respect to the input:\n\n```python\nimport torch\nimport torch.nn as nn\n\nstack = nn.Sequential(nn.Conv2d(1, 1, 3, padding=1), nn.ReLU(),\n                      nn.Conv2d(1, 1, 3, padding=1))\nfor conv in (stack[0], stack[2]):              # set the weights to 1 so every connection shows up\n    nn.init.ones_(conv.weight)\n    nn.init.zeros_(conv.bias)\n\nx = torch.rand(1, 1, 9, 9, requires_grad=True)\nstack(x)[0, 0, 4, 4].backward()                # look only at the single center output\n\nprint(x.grad[0, 0].int())\n```\n\n```text\ntensor([[0, 0, 0, 0, 0, 0, 0, 0, 0],\n        [0, 0, 0, 0, 0, 0, 0, 0, 0],\n        [0, 0, 1, 2, 3, 2, 1, 0, 0],\n        [0, 0, 2, 4, 6, 4, 2, 0, 0],\n        [0, 0, 3, 6, 9, 6, 3, 0, 0],\n        [0, 0, 2, 4, 6, 4, 2, 0, 0],\n        [0, 0, 1, 2, 3, 2, 1, 0, 0],\n        [0, 0, 0, 0, 0, 0, 0, 0, 0],\n        [0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=torch.int32)\n```\n\nThe nonzero region is exactly $5\\times5$. The numbers in the grid count “how many paths lead from this input point to the output”: 9 at the center, just 1 at the corners. Two $3\\times3$ layers see as far as a $5\\times5$; positions near the middle just carry more weight.\n\nThis replacement raises two questions. First, does it lose expressiveness? Second, since what we're factorizing is the linear $5\\times5$ operation, shouldn't the first layer stay linear, without a ReLU?\n\nThe authors ran several control experiments. One compares two Inception models: one uses a linear activation in the first factorized layer, the other uses ReLU in both layers. On the validation set they settle at 76.2% and 77.2% top-1 accuracy, respectively. The authors say the linear version was worse in all their control experiments; their explanation is that the extra nonlinearity lets the network learn more variations, especially with batch normalization.\n\n### Splitting Further into 1×n and n×1\n\nBy this logic, any kernel larger than $3\\times3$ can be replaced with a chain of $3\\times3$ layers. Can $3\\times3$ itself be split further?\n\nSplitting it into two $2\\times2$ layers saves only 11%. The authors found a better split: asymmetric convolutions. Do a $3\\times1$ first, then a $1\\times3$, and the field of view is still $3\\times3$.\n\n[![A diagram narrow at the top and wide at the bottom: a 3×3 input region at the bottom, a 3×1 strip of cells in the middle, and a single output cell at the top; red lines show the output first connecting to the 3 middle cells, each of which in turn connects to a whole row of the input](/content/papers/assets/26d9fd5f020cc9ce53cac89b03dbc710d155db812e4745d2e1c47a456faa3b1a.png)](/content/papers/assets/26d9fd5f020cc9ce53cac89b03dbc710d155db812e4745d2e1c47a456faa3b1a.png)\n\n*Figure 3 from the paper, excerpted from [**page 4 of the v3 PDF**](https://arxiv.org/pdf/1512.00567v3#page=4). The mini-network that replaces a $3\\times3$ convolution: the lower layer is a $3\\times1$ convolution with 3 outputs.*\n\nCheck it the same way:\n\n```python\nimport torch\nimport torch.nn as nn\n\npair = nn.Sequential(nn.Conv2d(1, 1, (1, 3), padding=(0, 1)),   # 1×3: looks only left and right\n                     nn.Conv2d(1, 1, (3, 1), padding=(1, 0)))   # 3×1: looks only up and down\nfor conv in pair:\n    nn.init.ones_(conv.weight)\n    nn.init.zeros_(conv.bias)\n\nx = torch.rand(1, 1, 7, 7, requires_grad=True)\npair(x)[0, 0, 3, 3].backward()\nprint(x.grad[0, 0].int())\n\nfor name, cost in [(\"3×3\", 9), (\"1×3 + 3×1\", 3 + 3), (\"two 2×2\", 4 + 4),\n                   (\"7×7\", 49), (\"1×7 + 7×1\", 7 + 7)]:\n    print(f\"{name}: {cost}\")\n```\n\n```text\ntensor([[0, 0, 0, 0, 0, 0, 0],\n        [0, 0, 0, 0, 0, 0, 0],\n        [0, 0, 1, 1, 1, 0, 0],\n        [0, 0, 1, 1, 1, 0, 0],\n        [0, 0, 1, 1, 1, 0, 0],\n        [0, 0, 0, 0, 0, 0, 0],\n        [0, 0, 0, 0, 0, 0, 0]], dtype=torch.int32)\n3×3: 9\n1×3 + 3×1: 6\ntwo 2×2: 8\n7×7: 49\n1×7 + 7×1: 14\n```\n\nThe lines at the bottom give the number of multiplications per output, per pair of input and output channels. $1\\times3$ plus $3\\times1$ takes 6, a third fewer than $3\\times3$; two $2\\times2$ layers take 8, only a ninth fewer. The larger the kernel, the bigger the saving: $7\\times7$ needs 49, while $1\\times7$ plus $7\\times1$ needs only 14.\n\nThis trick doesn't work well everywhere, though. The authors found that it does poorly early in the network but works very well on medium-sized feature maps (sides of roughly 12 to 20). The actual network uses $1\\times7$ followed by $7\\times1$ on $17\\times17$ feature maps:\n\n[![Diagram of an Inception module: Base at the bottom splits upward into four branches. The leftmost is a 1x1 followed by four layers, 1xn, nx1, 1xn, and nx1; the second is a 1x1 followed by 1xn and nx1; the third is Pool followed by 1x1; the rightmost is just a 1x1. The four branches merge at Filter Concat](/content/papers/assets/d7b13cfd7a93816701d9571a2e87bd67983a5e6ffe9a7b93cbcf40e82db76a2f.png)](/content/papers/assets/d7b13cfd7a93816701d9571a2e87bd67983a5e6ffe9a7b93cbcf40e82db76a2f.png)\n\n*Figure 6 from the paper, excerpted from [**page 5 of the v3 PDF**](https://arxiv.org/pdf/1512.00567v3#page=5). An Inception module after factorizing the $n\\times n$ convolutions; on $17\\times17$ feature maps, the authors use $n=7$.*\n\nCompare it with GoogLeNet's original module, which had four branches: $1\\times1$, $3\\times3$, $5\\times5$, and pooling. Here the leftmost branch amounts to two $7\\times7$ layers, each split into a pair of asymmetric convolutions; the second branch is the split version of a single $7\\times7$ layer. The $35\\times35$ stage uses the module from Figure 5, which replaces the $5\\times5$ with two $3\\times3$ layers.\n\nAt the smallest feature maps, $8\\times8$, the authors use the idea differently:\n\n[![Diagram of an Inception module: the leftmost branch is 1x1 then 3x3, which then splits into two side-by-side convolutions, 1x3 and 3x1; the second branch is a 1x1 that splits into 1x3 and 3x1; there is also Pool followed by 1x1, and a lone 1x1. All outputs merge at Filter Concat](/content/papers/assets/1475f4de3132dd0d1dbd081ddc453fb140796cb802987fa051a70356070b1b41.png)](/content/papers/assets/1475f4de3132dd0d1dbd081ddc453fb140796cb802987fa051a70356070b1b41.png)\n\n*Figure 7 from the paper, excerpted from [**page 5 of the v3 PDF**](https://arxiv.org/pdf/1512.00567v3#page=5). An Inception module with expanded filter outputs, used only on the coarsest, $8\\times8$ feature maps.*\n\nNote that $1\\times3$ and $3\\times1$ are **side by side** here: both outputs go straight into the final concatenation instead of being chained. This increases the number of output channels, in line with principle 2: producing high-dimensional representations on the last feature maps. The authors use it only at $8\\times8$, reasoning that at this stage the local processing done by $1\\times1$ convolutions matters more than spatial aggregation.\n\n### What Auxiliary Classifiers Really Do\n\nGoogLeNet attached two auxiliary classifiers to the middle of the network. The original idea was to send useful gradients to the earlier layers, fighting vanishing gradients and speeding up convergence.\n\nWhen the authors took another look, that's not what they found. Early in training, the curves with and without auxiliary classifiers are nearly identical; only near the end of training does the network with auxiliary classifiers pull ahead, settling at a slightly higher plateau. Removing the lower of the two auxiliary classifiers didn't hurt the final quality either.\n\nSo GoogLeNet's original hypothesis, that the auxiliary branches help the low-level features evolve, is most likely mistaken in the authors' view. Their new explanation: auxiliary classifiers act as **regularizers**. The evidence is that the main classifier performs better when the auxiliary branch has batch normalization or dropout; adding batch normalization to the auxiliary branch gives a 0.4% absolute gain in top-1 accuracy. Inception-v3's auxiliary classifier sits on top of the last $17\\times17$ feature map.\n\nThe authors also remark in passing that this offers weak supporting evidence for the conjecture that batch normalization acts as a regularizer.\n\n### Shrinking the Feature Map Without Choking the Information\n\nConvolutional networks usually shrink feature maps with pooling. Suppose we want to turn $35\\times35\\times320$ into $17\\times17\\times640$. There are two direct ways to do it:\n\n- **Convolve, then pool**: first raise the channels from 320 to 640 at $35\\times35$, then pool. No information gets choked, but the most expensive convolution runs on the large feature map.\n- **Pool, then convolve**: first pool to $17\\times17$, then add channels. Much cheaper, but along the way there's a step with only $17\\times17\\times320$, which violates principle 1.\n\nLet's do a rough calculation with a single $3\\times3$ convolution:\n\n```python\nd, k = 35, 320                               # 35×35×320, to become 17×17×640\nh = 17\n\nconv_then_pool = d * d * k * (2 * k) * 9     # first a 320→640 3×3 convolution on 35×35\npool_then_conv = h * h * k * (2 * k) * 9     # first pool to 17×17, then 320→640\nparallel = h * h * k * k * 9                 # the stride-2 convolution outputs only 320 channels; pooling supplies the other half\n\nfor name, macs in [(\"Convolve, then pool\", conv_then_pool), (\"Pool, then convolve\", pool_then_conv), (\"Convolve and pool in parallel\", parallel)]:\n    print(f\"{name}:{macs / 1e6:7.1f} M multiply-adds\")\n\nprint(\"Pooling first leaves only\", h * h * k, \"values in between; the input has\", d * d * k, \"values\")\n```\n\n```text\nConvolve, then pool: 2257.9 M multiply-adds\nPool, then convolve:  532.7 M multiply-adds\nConvolve and pool in parallel:  266.3 M multiply-adds\nPooling first leaves only 92480 values in between; the input has 392000 values\n```\n\nConvolving first costs more than four times as much as pooling first.\n\nThe authors' solution is the third line: put a stride-2 convolution and a stride-2 pooling **side by side**, each shrinking the feature map directly, then concatenate the channels from both:\n\n[![Two diagrams side by side. The left shows the module's operations: Base splits upward into three branches; two are a 1x1 followed by a stride-2 3x3 convolution (one of them with an extra stride-1 3x3 in front), and the third is stride-2 pooling; they merge at Filter Concat. The right shows it in terms of feature map sizes: 35x35x320 splits into a conv side and a pool side, each producing 17x17x320, which are concatenated into 17x17x640](/content/papers/assets/9d6b75a1dd143d0b517f52adf42c4fe794cb400e3276a8109561db2277fb2fb6.png)](/content/papers/assets/9d6b75a1dd143d0b517f52adf42c4fe794cb400e3276a8109561db2277fb2fb6.png)\n\n*Figure 10 from the paper, excerpted from [**page 6 of the v3 PDF**](https://arxiv.org/pdf/1512.00567v3#page=6). Left: an Inception module that shrinks the feature map while increasing the channels; right: the same thing in terms of feature map sizes.*\n\nThe diagram on the right is the easiest to read: $35\\times35\\times320$ splits into two paths. The convolution path outputs $17\\times17\\times320$, the pooling path also outputs $17\\times17\\times320$, and together they make $17\\times17\\times640$. The convolution only has to output half the channels, so it costs half as much as even the pool-first approach; and the output has 640 channels from the start, without passing through the squeezed step.\n\n### Putting the Pieces Together\n\nThe authors assemble these components into a new network:\n\n| Type | Kernel/stride or note | Input size |\n| --- | --- | --- |\n| conv | $3\\times3$/2 | $299\\times299\\times3$ |\n| conv | $3\\times3$/1 | $149\\times149\\times32$ |\n| conv (padded) | $3\\times3$/1 | $147\\times147\\times32$ |\n| pool | $3\\times3$/2 | $147\\times147\\times64$ |\n| conv | $3\\times3$/1 | $73\\times73\\times64$ |\n| conv | $3\\times3$/2 | $71\\times71\\times80$ |\n| conv | $3\\times3$/1 | $35\\times35\\times192$ |\n| 3 × Inception | As in Figure 5 | $35\\times35\\times288$ |\n| 5 × Inception | As in Figure 6 | $17\\times17\\times768$ |\n| 2 × Inception | As in Figure 7 | $8\\times8\\times1280$ |\n| pool | $8\\times8$ | $8\\times8\\times2048$ |\n| linear | logits | $1\\times1\\times2048$ |\n| softmax | classifier | $1\\times1\\times1000$ |\n\n*Table: Excerpted from Table 1 of the paper. Each row's input size is the output size of the row above.*\n\nThe first three $3\\times3$ convolutions come from splitting up the traditional $7\\times7$ first layer, the same idea as replacing $5\\times5$ with two $3\\times3$ layers. Between the three Inception stages, modules like the one in Figure 10 shrink the feature maps. The whole network is 42 layers deep.\n\nYou might wonder why the input is such an odd number, $299\\times299$. Let's work through the table's settings:\n\n```python\ndef out(n, k, s, pad=0):\n    return (n + 2 * pad - k) // s + 1\n\nn = 299\nfor name, k, s, pad in [(\"conv 3×3/2\", 3, 2, 0), (\"conv 3×3/1\", 3, 1, 0), (\"conv 3×3/1 padded\", 3, 1, 1),\n                        (\"pool 3×3/2\", 3, 2, 0), (\"conv 3×3/1\", 3, 1, 0), (\"conv 3×3/2\", 3, 2, 0)]:\n    m = out(n, k, s, pad)\n    print(f\"{name}: {n} → {m}, remainder {(n + 2 * pad - k) % s}\")\n    n = m\n\nfor name in [\"reduce to 17×17\", \"reduce to 8×8\"]:\n    m = out(n, 3, 2)\n    print(f\"{name}: {n} → {m}, remainder {(n - 3) % 2}\")\n    n = m\n```\n\n```text\nconv 3×3/2: 299 → 149, remainder 0\nconv 3×3/1: 149 → 147, remainder 0\nconv 3×3/1 padded: 147 → 147, remainder 0\npool 3×3/2: 147 → 73, remainder 0\nconv 3×3/1: 73 → 71, remainder 0\nconv 3×3/2: 71 → 35, remainder 0\nreduce to 17×17: 35 → 17, remainder 0\nreduce to 8×8: 17 → 8, remainder 0\n```\n\nEvery stride-2 layer without padding divides evenly, so not a single row of pixels is dropped, and the sizes land exactly on the table's $35$, $17$, and $8$. The paper doesn't explain why it chose 299, but this number does make the whole path line up neatly.\n\n### Keeping the Model from Getting Too Sure of Itself\n\nThe last component has nothing to do with the architecture; it deals with the labels.\n\nFor a training image $x$, the model computes the probability of each class with a softmax:\n\n$$\np(k\\mid x) = \\frac{\\exp(z_k)}{\\sum_{i=1}^{K}\\exp(z_i)}\n$$\n\n$z_k$ are the logits. Writing the ground-truth label distribution as $q(k\\mid x)$, the cross-entropy loss is:\n\n$$\n\\ell = -\\sum_{k=1}^{K} \\log p(k)\\, q(k)\n$$\n\nIts gradient with respect to the logits is simple: $\\partial\\ell/\\partial z_k = p(k) - q(k)$, which lies between $-1$ and $1$.\n\nOrdinary training uses one-hot labels: $q(y)=1$ for the correct class $y$ and 0 for everything else. To minimize the loss, $p(y)$ has to equal 1, which is impossible with finite logits, so the only option is for $z_y$ to keep pulling further ahead of the other logits.\n\nThe authors point out two problems with this. First, the model learns to assign full confidence to every training image, which may not generalize. Second, the gap between the largest logit and the others gets pushed very wide, and since the gradient is bounded, the model has a harder time adapting. Put plainly, the model is too sure of its own predictions.\n\nTheir fix is to replace the label distribution:\n\n$$\nq'(k\\mid x) = (1-\\epsilon)\\,\\delta_{k,y} + \\epsilon\\, u(k)\n$$\n\n$u(k)$ is a fixed distribution independent of the image; the authors use the uniform distribution $u(k)=1/K$. This is called label smoothing regularization (LSR). ImageNet has $K=1000$ classes, and $\\epsilon=0.1$.\n\nLet's treat the logits directly as parameters and see where each kind of label pushes them:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\nK, eps, y = 1000, 0.1, 0\nhard = torch.zeros(K); hard[y] = 1\nsmooth = (1 - eps) * hard + eps / K\nprint(\"Smoothed target: correct class\", round(smooth[y].item(), 4), \"and each other class\", round(smooth[1].item(), 4))\n\nfor name, q in [(\"one-hot\", hard), (\"label smoothing\", smooth)]:\n    z = torch.zeros(K, requires_grad=True)               # treat the logits directly as parameters and see where the loss pushes them\n    opt = torch.optim.SGD([z], lr=1.0)\n    for step in range(1, 20001):\n        loss = -(q * F.log_softmax(z, 0)).sum()\n        opt.zero_grad(); loss.backward(); opt.step()\n        if step in (100, 1000, 20000):\n            gap = (z[y] - z[1:].max()).item()\n            print(f\"{name:15s} step {step:5d}: correct-class logit leads by {gap:6.3f}\")\n\nprint(\"label smoothing endpoint: ln(0.9001 / 0.0001) =\", round(torch.log(torch.tensor(0.9001 / 0.0001)).item(), 3))\n```\n\n```text\nSmoothed target: correct class 0.9001 and each other class 0.0001\none-hot         step   100: correct-class logit leads by 11.417\none-hot         step  1000: correct-class logit leads by 13.805\none-hot         step 20000: correct-class logit leads by 16.810\nlabel smoothing step   100: correct-class logit leads by  9.105\nlabel smoothing step  1000: correct-class logit leads by  9.105\nlabel smoothing step 20000: correct-class logit leads by  9.105\nlabel smoothing endpoint: ln(0.9001 / 0.0001) = 9.105\n```\n\nWith one-hot labels, the gap keeps growing: the longer you train, the further it's pushed. With smoothed labels, the gap stops at 9.105. You can compute this number directly: at the minimum of the loss, $p = q'$, and under a softmax the difference between two classes' logits equals the log of their probability ratio, which is $\\ln(0.9001/0.0001)$.\n\nThe authors also offer another way to look at it. Expand the cross-entropy:\n\n$$\nH(q', p) = (1-\\epsilon)\\,H(q, p) + \\epsilon\\, H(u, p)\n$$\n\nThis amounts to adding a penalty on top of the original loss: if the predicted distribution $p$ strays too far from the uniform distribution $u$, it pays a price.\n\nOn ILSVRC 2012, adding LSR lowers top-1 error from 23.1% to 22.8% and top-5 from 6.3% to 6.1%.\n\n\n> [!NOTE]\n> \n> Four years later, Rafael Müller, Simon Kornblith, and Geoffrey Hinton revisited the technique in [When Does Label Smoothing Help?](https://arxiv.org/abs/1906.02629) at NeurIPS 2019. They found that label smoothing improves model calibration and also makes the penultimate-layer representations of training examples from the same class cluster more tightly; the cost is that the logits carry less information about “how similar this image is to other classes,” so teacher models trained with label smoothing do much worse at knowledge distillation.\n\n\n### Training Setup\n\nThe authors train with TensorFlow's distributed system, using 50 replicas, each running on one NVIDIA Kepler GPU with a batch size of 32, for 100 epochs.\n\nEarly experiments used momentum; the best models switched to RMSProp (decay 0.9, $\\epsilon=1.0$). The learning rate is 0.045, multiplied by 0.94 every two epochs. Gradient clipping at 2.0 is used to stabilize training, and evaluation uses a running average of the parameters.\n\n## Discussion\n\n### Adding One Change at a Time\n\n| Network | Top-1 error | Top-5 error | Cost (billions of operations) |\n| --- | ---: | ---: | ---: |\n| GoogLeNet | 29% | 9.2% | 1.5 |\n| BN-GoogLeNet | 26.8% | – | 1.5 |\n| BN-Inception | 25.2% | 7.8% | 2.0 |\n| Inception-v2 | 23.4% | – | 3.8 |\n| + RMSProp | 23.1% | 6.3% | 3.8 |\n| + Label Smoothing | 22.8% | 6.1% | 3.8 |\n| + Factorized 7×7 | 21.6% | 5.8% | 4.8 |\n| + BN auxiliary classifier | **21.2%** | **5.6%** | 4.8 |\n\n*Table: Excerpted from Table 3 of the paper. ILSVRC 2012 validation set, single crop. From Inception-v2 down, each row includes all the changes above it.*\n\nThis table is cumulative: each row adds one more change on top of the row before. The last row, “BN auxiliary classifier,” means the fully connected layer in the auxiliary classifier also gets batch normalization, not just the convolutional layers. **The version with every change added is what the authors call Inception-v3.**\n\nA few things worth noting:\n\n- Switching to the new architecture alone (the Inception-v2 row) lowers top-1 error from BN-Inception's 25.2% to 23.4%, but computation also rises from 2.0 to 3.8.\n- RMSProp and label smoothing add no computation, and each gains about 0.3 percentage points of top-1.\n- Splitting the first-layer $7\\times7$ into three $3\\times3$ layers raises computation from 3.8 to 4.8 and improves top-1 by 1.2 percentage points, the biggest step among the later changes.\n\nOverall, Inception-v3 costs about 2.4 times as much computation as BN-Inception and 3.2 times as much as GoogLeNet.\n\n\n> [!TIP]\n> \n> The naming here is a bit messy. This paper calls the new architecture Inception-v2, and only with all the changes added does it become Inception-v3. But the following year's Inception-v4 paper, looking back on the family history, calls BN-Inception, the version with batch normalization added, Inception-v2. Whenever you see “Inception-v2” from now on, check which one is meant.\n> \n> For where BN-Inception comes from, see our note:\n> \n> - **[[15.02] BatchNorm: Batch Normalization](/blog/papers/normalization/batchnorm)**\n\n\nAll evaluations use the 48,238 non-blacklisted images of the ILSVRC 2012 validation set. The authors also tried all 50,000 images; the results differed by about 0.1 percentage points in top-5 error and about 0.2 in top-1.\n\n### How Much Does Lower Resolution Cost?\n\nDetection systems often have to classify small patches of an image, where objects are small and blurry. The common belief is that higher input resolution is better, but the authors point out that two things are mixed up here: the resolution of the first layer's receptive field, and the model's own capacity and computation. If you just shrink the input and leave the model alone, you're using a much cheaper model to solve a harder problem.\n\nSo they hold computation fixed and change only the input resolution. When the input gets smaller, they remove early strides or pooling so the later feature maps stay about the same size:\n\n| Receptive field size | Adjustment | Top-1 accuracy |\n| --- | --- | ---: |\n| $79\\times79$ | First layer stride 1, no pooling after | 75.2% |\n| $151\\times151$ | First layer stride 1, then max pooling | 76.4% |\n| $299\\times299$ | First layer stride 2, then max pooling | 76.6% |\n\n*Table: Excerpted from Table 2 of the paper; the adjustments are summarized from the experiment description in Section 9.*\n\nThe three networks have almost the same computational cost. The lower-resolution networks take longer to train, but their final results aren't far from $299\\times299$: $151\\times151$ trails by only 0.2 percentage points, and $79\\times79$ by 1.4.\n\nFrom this, the authors suggest that in detection pipelines like R-CNN, it may be worth using dedicated networks for small objects that are low-resolution but still computationally heavy.\n\n### Comparison with Other Models\n\nSingle model, multiple crops:\n\n| Network | Crops | Top-1 error | Top-5 error |\n| --- | ---: | ---: | ---: |\n| GoogLeNet | 144 | – | 7.89% |\n| VGG | – | 24.4% | 6.8% |\n| BN-Inception | 144 | 22% | 5.82% |\n| PReLU | – | 21.59% | 5.71% |\n| Inception-v3 | 12 | 19.47% | 4.48% |\n| **Inception-v3** | **144** | **18.77%** | **4.2%** |\n\n*Table: Selected rows excerpted from Table 4 of the paper.*\n\nMulti-model ensembles:\n\n| Network | Models | Crops | Top-1 error | Top-5 error |\n| --- | ---: | ---: | ---: | ---: |\n| VGGNet | 2 | – | 23.7% | 6.8% |\n| GoogLeNet | 7 | 144 | – | 6.67% |\n| PReLU | – | – | – | 4.94% |\n| BN-Inception | 6 | 144 | 20.1% | 4.9% |\n| **Inception-v3** | **4** | **144** | **17.2%** | **3.58%** |\n\n*Table: Excerpted from Table 5 of the paper. Inception-v3's 3.58% is a test-set result; everything else comes from the validation set.*\n\nIn the conclusion, the authors compare against the PReLU network of He et al.: Inception-v3 cuts top-5 and top-1 error by 25% and 14% relative, while being six times cheaper computationally and using at least five times fewer parameters (which the authors mark as an estimate). The four-model ensemble's top-5 error is about half that of the winning 2014 GoogLeNet ensemble.\n\n## Conclusion\n\nThis paper reads more like a design notebook. Table 3 adds changes one at a time, so you can see what each one contributes, and the authors candidly label their four design principles as rules of thumb that still need verification.\n\nTo wrap up, here are the components this paper left behind:\n\n1. **Split large kernels into small ones**: $5\\times5$ becomes two $3\\times3$ layers, and $n\\times n$ becomes $1\\times n$ followed by $n\\times1$.\n2. **When shrinking feature maps, run convolution and pooling side by side**, so information doesn't get choked.\n3. **Auxiliary classifiers act more like regularizers.**\n4. **Label smoothing**, so the model isn't too sure of its answers.\n\nIf you can take only one, it's probably label smoothing: it's independent of the architecture, takes a few lines of code, and appears in [ConvNeXt](/blog/papers/classic-cnns/convnext)'s training recipe too. The next version of Inception meets ResNet; we'll pick it up in the [next note](/blog/papers/classic-cnns/inception-v4).\n","imageSizes":{"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png":{"width":419,"height":358},"/content/papers/assets/26d9fd5f020cc9ce53cac89b03dbc710d155db812e4745d2e1c47a456faa3b1a.png":{"width":293,"height":401},"/content/papers/assets/d7b13cfd7a93816701d9571a2e87bd67983a5e6ffe9a7b93cbcf40e82db76a2f.png":{"width":623,"height":824},"/content/papers/assets/1475f4de3132dd0d1dbd081ddc453fb140796cb802987fa051a70356070b1b41.png":{"width":625,"height":539},"/content/papers/assets/9d6b75a1dd143d0b517f52adf42c4fe794cb400e3276a8109561db2277fb2fb6.png":{"width":635,"height":365}}}