{"notes":[{"id":"classic-cnns/nin","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NiN","subtitle":"Foreshadowing the 1×1","author":"Z. Yuan","paperDate":"2013-12-16","publishedOn":"2025-09-24","world":"external/network-in-network","paperTitle":"Network In Network","paperUrl":"https://arxiv.org/abs/1312.4400","bibliography":{"title":"Network In Network","authors":["Min Lin","Qiang Chen","Shuicheng Yan"],"url":"https://arxiv.org/abs/1312.4400","pdfUrl":"https://arxiv.org/pdf/1312.4400v3","codeUrl":null,"preprint":{"id":"1312.4400","submittedOn":"2013-12-16","url":"https://arxiv.org/abs/1312.4400"},"publication":null,"citation":"@misc{classiccnnsnin2013,\n  title = {{Network In Network}},\n  author = {Min Lin and Qiang Chen and Shuicheng Yan},\n  year = {2013},\n  eprint = {1312.4400},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.4400}\n}\n","citationPath":"/content/papers/citations/classic-cnns/nin.db58639ac47c.bib"},"thumbnail":{"src":"/content/papers/assets/8024fc245eb852a2c0cd34b59c18c680b92766c33d621b934dbf102c1cacda13.png","width":896,"height":330},"locale":"en","path":"/content/papers/en/classic-cnns/nin.55401b294dee.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"running-a-small-mlp-on-every-patch","text":"Running a Small MLP on Every Patch","depth":3},{"id":"replacing-the-fully-connected-layers-with-an-average","text":"Replacing the Fully Connected Layers with an Average","depth":3},{"id":"the-whole-network","text":"The Whole Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"cifar-10-and-cifar-100","text":"CIFAR-10 and CIFAR-100","depth":3},{"id":"svhn-and-mnist","text":"SVHN and MNIST","depth":3},{"id":"does-global-average-pooling-really-regularize","text":"Does Global Average Pooling Really Regularize?","depth":3},{"id":"did-the-last-layer-really-become-confidence-maps","text":"Did the Last Layer Really Become Confidence Maps?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/cam","relation":"referenced","topic":""}]},{"id":"classic-cnns/inception-v1","category":"classic-cnns","legacyPath":"classic-cnns/1409-inception-v1","legacySlug":"1409-inception-v1","legacyPaths":[],"title":"GoogLeNet","subtitle":"Also Known as Inception-V1","author":"Z. Yuan","paperDate":"2014-09-17","publishedOn":"2024-09-22","paperTitle":"Going Deeper with Convolutions","paperUrl":"https://arxiv.org/abs/1409.4842","bibliography":{"title":"Going Deeper with Convolutions","authors":["Christian Szegedy","Wei Liu","Yangqing Jia","Pierre Sermanet","Scott Reed","Dragomir Anguelov","Dumitru Erhan","Vincent Vanhoucke","Andrew Rabinovich"],"url":"https://arxiv.org/abs/1409.4842","pdfUrl":"https://arxiv.org/pdf/1409.4842","codeUrl":null,"preprint":{"id":"1409.4842","submittedOn":"2014-09-17","url":"https://arxiv.org/abs/1409.4842"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2015.7298594"},"citation":"@misc{classiccnnsinceptionv12014,\n  title = {{Going Deeper with Convolutions}},\n  author = {Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich},\n  year = {2014},\n  eprint = {1409.4842},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1409.4842}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v1.3d20e0e32bcd.bib"},"thumbnail":{"src":"/content/papers/assets/254b970b1374c24267b60c2d684338760e9ba8b10dad761a5fa15f1bb8bcf6ef.jpg","width":1330,"height":394},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v1.41dca1189319.md","headings":[{"id":"problem-solved","text":"Problem Solved","depth":2},{"id":"the-inception-module","text":"The Inception Module","depth":3},{"id":"configuration-parameters","text":"Configuration Parameters","depth":3},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"performance-at-ilsvrc-2014","text":"Performance at ILSVRC 2014","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnext","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"Convolutional with 100 Layers","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"en","path":"/content/papers/en/classic-cnns/resnet.b65375cb9846.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-is-residual","text":"What is Residual?","depth":3},{"id":"residual-learning","text":"Residual Learning","depth":3},{"id":"residual-module","text":"Residual Module","depth":3},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"architectures-and-computation","text":"Architectures and Computation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"deeper-bottleneck-architectures","text":"Deeper Bottleneck Architectures","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"Training deeper convolutional networks"},{"id":"classic-cnns/densenet","relation":"next","topic":"Training deeper convolutional networks"}]},{"id":"classic-cnns/cam","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"CAM","subtitle":"Where Is the Model Looking?","author":"Z. Yuan","paperDate":"2015-12-14","publishedOn":"2026-01-05","world":"external/cam","paperTitle":"Learning Deep Features for Discriminative Localization","paperUrl":"https://arxiv.org/abs/1512.04150","bibliography":{"title":"Learning Deep Features for Discriminative Localization","authors":["Bolei Zhou","Aditya Khosla","Agata Lapedriza","Aude Oliva","Antonio Torralba"],"url":"https://arxiv.org/abs/1512.04150","pdfUrl":"https://arxiv.org/pdf/1512.04150v1","codeUrl":null,"preprint":{"id":"1512.04150","submittedOn":"2015-12-14","url":"https://arxiv.org/abs/1512.04150"},"publication":null,"citation":"@misc{classiccnnscam2015,\n  title = {{Learning Deep Features for Discriminative Localization}},\n  author = {Bolei Zhou and Aditya Khosla and Agata Lapedriza and Aude Oliva and Antonio Torralba},\n  year = {2015},\n  eprint = {1512.04150},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.04150}\n}\n","citationPath":"/content/papers/citations/classic-cnns/cam.2ce837236f9d.bib"},"thumbnail":{"src":"/content/papers/assets/c3c6af7f1943634784331ca03a55efc3e68a7714d936d953a14f11b0828d9cf8.png","width":1372,"height":615},"locale":"en","path":"/content/papers/en/classic-cnns/cam.5e1bd71357c2.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"splitting-the-score-back-over-space","text":"Splitting the Score Back Over Space","depth":3},{"id":"average-vs-max-whats-the-difference","text":"Average vs. Max: What's the Difference?","depth":3},{"id":"raising-the-resolution-of-the-last-layer","text":"Raising the Resolution of the Last Layer","depth":3},{"id":"from-heat-map-to-box","text":"From Heat Map to Box","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-much-does-classification-suffer-without-the-fully-connected-layers","text":"How Much Does Classification Suffer Without the Fully Connected Layers?","depth":3},{"id":"weakly-supervised-localization","text":"Weakly Supervised Localization","depth":3},{"id":"moving-to-other-tasks","text":"Moving to Other Tasks","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/nin","relation":"referenced","topic":""},{"id":"classic-cnns/zfnet","relation":"referenced","topic":""},{"id":"classic-cnns/resnet","relation":"referenced","topic":""}]}],"text":"In the 2013 ImageNet classification challenge, the team from the National University of Singapore took second place. Its members included Min Lin and Qiang Chen.\n\nIn December that year, the two of them, together with Shuicheng Yan, released this paper, which was later published at ICLR 2014. The paper itself has no ImageNet experiments and compares only on small datasets such as CIFAR, SVHN, and MNIST. Yet the two components it left behind turned up all over convolutional networks in the years that followed: $1\\times1$ convolutions and global average pooling.\n\nBoth components grew out of the same question: are the filters in a convolutional network too simple?\n\n## Defining the Problem\n\nFirst, look at what an ordinary convolutional layer does. Taking ReLU as an example, the output of the $k$-th feature map at position $(i, j)$ is:\n\n$$\nf_{i,j,k} = \\max(w_k^T x_{i,j},\\ 0)\n$$\n\n$x_{i,j}$ is the input patch centered at $(i, j)$, and $w_k$ is the $k$-th filter. In other words, each filter takes one linear inner product with the patch and then applies ReLU once. The authors call this a generalized linear model (GLM).\n\nThere's only so much a linear model can separate. Suppose a filter is meant to detect “diagonal edges,” and the appearances of diagonal edges under different lighting and thicknesses happen to all fall on the same side of one hyperplane; then a single filter is enough. But the many appearances of one concept are often spread over a curved manifold, and one hyperplane can't cut them out cleanly.\n\nHow does an ordinary convolutional network cope? It adds more filters, each responsible for one appearance. The authors point out that this pushes the burden onto the next layer: the next layer has to consider combinations of all the appearances from the layer below before it can piece together higher-level concepts.\n\nThe maxout network, published shortly before, took a step forward. It takes the maximum across several affine feature maps:\n\n$$\nf_{i,j,k} = \\max_m\\,(w_{k_m}^T x_{i,j})\n$$\n\nTaking the maximum of several linear functions gives a piecewise linear convex function, so it can separate concepts that lie within convex sets. But the authors consider “concepts lie within convex sets” to be an assumption that still may not hold.\n\nSo could we place, on each local patch, a model that makes no such assumption and can approximate general functions?\n\nThere's another problem at the end of a convolutional network. The last few layers are usually fully connected, and the authors see them as something of a black box: it's hard to say how category information gets passed back to the convolutional layers. Fully connected layers also overfit easily and have to be propped up by dropout.\n\n## Solving the Problem\n\n### Running a Small MLP on Every Patch\n\nThe authors' answer is to replace the linear filter with a small multilayer perceptron (MLP), which likewise slides over every position of the input, with all positions sharing the same weights. This kind of layer is called mlpconv:\n\n[![Two diagrams side by side. (a) Linear convolution layer: a local patch of the input passes through one linear filter to produce one point on the output feature map. (b) mlpconv layer: the same local patch first passes through a small two-layer fully connected network, then produces one point on the output](/content/papers/assets/8024fc245eb852a2c0cd34b59c18c680b92766c33d621b934dbf102c1cacda13.png)](/content/papers/assets/8024fc245eb852a2c0cd34b59c18c680b92766c33d621b934dbf102c1cacda13.png)\n\n*Figure 1 from the paper, excerpted from [**page 2 of the v3 PDF**](https://arxiv.org/pdf/1312.4400v3#page=2). (a) A linear convolution layer; (b) an mlpconv layer, which adds a small MLP over the patch.*\n\nWith an $n$-layer MLP, mlpconv computes:\n\n$$\n\\begin{aligned}\nf^{1}_{i,j,k_1} &= \\max\\left({w^{1}_{k_1}}^T x_{i,j} + b_{k_1},\\ 0\\right) \\\\\n&\\ \\ \\vdots \\\\\nf^{n}_{i,j,k_n} &= \\max\\left({w^{n}_{k_n}}^T f^{n-1}_{i,j} + b_{k_n},\\ 0\\right)\n\\end{aligned}\n$$\n\nThe first line is exactly the same as an ordinary convolution: an inner product with the whole patch $x_{i,j}$. From the second line on, the input becomes the previous layer's output at **the same position** $(i, j)$, $f^{n-1}_{i,j}$; in other words, only the values of all channels at this one position.\n\nLooking at a single position and taking a weighted combination of all its channels is exactly what a $1\\times1$ convolution does. The authors state this equivalence directly: mlpconv equals one ordinary convolutional layer followed by several $1\\times1$ convolutional layers.\n\nLet's verify it once in PyTorch. The input is a 2-channel $5\\times5$ feature map; the small MLP takes the $2\\times3\\times3=18$ numbers of each flattened $3\\times3$ patch and passes them through two layers, 18 → 4 → 3:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\nx = torch.randn(1, 2, 5, 5)            # 1 image, 2 channels, 5×5\n\n# Small MLP: each 3×3 patch is flattened into 2×3×3 = 18 numbers, then goes 18 → 4 → 3\nw1, b1 = torch.randn(4, 18), torch.randn(4)\nw2, b2 = torch.randn(3, 4), torch.randn(3)\n\n# Approach 1: slide the MLP over every 3×3 patch\npatches = F.unfold(x, kernel_size=3)            # [1, 18, 9]: 9 positions, 18 dims each\nh = F.relu(w1 @ patches + b1[:, None])          # [1, 4, 9]\ny_mlp = F.relu(w2 @ h + b2[:, None]).reshape(1, 3, 3, 3)\n\n# Approach 2: a 3×3 convolution followed by a 1×1 convolution\nh = F.relu(F.conv2d(x, w1.reshape(4, 2, 3, 3), b1))\ny_conv = F.relu(F.conv2d(h, w2.reshape(3, 4, 1, 1), b2))\n\nprint(y_mlp.shape, y_conv.shape)\nprint(\"Max difference:\", (y_mlp - y_conv).abs().max().item())\n```\n\n```text\ntorch.Size([1, 3, 3, 3]) torch.Size([1, 3, 3, 3])\nMax difference: 4.76837158203125e-07\n```\n\nThe two approaches produce outputs of the same shape, differing only by floating-point error.\n\nApproach 1 is closer to the authors' idea: at every position, a small network judges “is this a particular concept?” Approach 2 is what it looks like in implementation: a $3\\times3$ convolution first looks at the whole patch, and the $1\\times1$ convolutions after it only mix channels at each position. The authors also describe it from the channel side: each $1\\times1$ convolutional layer is a “cross-channel parametric pooling” that reweights and recombines the previous layer's feature maps and then applies ReLU, layer after layer.\n\n\n> [!TIP]\n> \n> Later, GoogLeNet used $1\\times1$ convolutions heavily in its Inception modules, but mainly for a different purpose: reducing the number of channels so that the $3\\times3$ and $5\\times5$ convolutions after them would be cheaper.\n> \n> - **[[14.09] GoogLeNet: Also Known as Inception-V1](/blog/papers/classic-cnns/inception-v1)**\n\n\nCompared with maxout, the difference is this: maxout can only approximate convex functions, while the MLP in mlpconv is a universal function approximator that can handle concept distributions of many more shapes. The authors' reason for choosing an MLP over another universal approximator, the radial basis network, is a practical one too: an MLP can be trained by backpropagation together with the convolutional network, and it can itself be made deep.\n\n### Replacing the Fully Connected Layers with an Average\n\nThe second component deals with the end of the network.\n\nAn ordinary convolutional network flattens the feature maps of the last convolutional layer, feeds them into a few fully connected layers, and then into a softmax. NiN does this instead: the last mlpconv layer outputs **exactly as many feature maps as there are classes**, each feature map is averaged over space, and the resulting vector goes straight into the softmax. This is called global average pooling (GAP).\n\nTake CIFAR-10: there are 10 classes, so the last layer outputs 10 feature maps:\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nfeature_maps = torch.randn(1, 10, 8, 8)    # last layer: one 8×8 feature map for each of the 10 classes\n\nlogits = feature_maps.mean(dim=(2, 3))      # global average pooling\nprint(logits.shape)\n\nfc_params = 10 * 8 * 8 * 10 + 10            # if a fully connected layer to 10 classes were used instead\nprint(\"Fully connected layer parameters:\", fc_params, \"vs. global average pooling parameters: 0\")\n```\n\n```text\ntorch.Size([1, 10])\nFully connected layer parameters: 6410 vs. global average pooling parameters: 0\n```\n\nThe $8\\times8$ here is just an illustrative size. In a small network the gap doesn't look like much, but in AlexNet it's obvious: its last three fully connected layers ($9216\\to4096\\to4096\\to1000$) have $9216\\times4096+4096\\times4096+4096\\times1000 = 58{,}621{,}952$ weights alone.\n\nThe authors list three advantages of GAP:\n\n1. **No parameters**, so this layer itself can't overfit.\n2. **Each feature map corresponds to one class** and can be read directly as that class's confidence map, which is easier to interpret than fully connected layers.\n3. **Spatial information is summed out**, making it less sensitive to translations of the input.\n\nPoint 2 comes with a precondition: the last layer has to actually be able to learn a feature map that stands for “cat.” The authors argue this relies on the stronger local modeling of mlpconv; with ordinary linear convolutions, the requirement may be too strict. We'll see this in the experiments later.\n\n### The Whole Network\n\n[![The overall NiN architecture: three mlpconv layers in sequence, each containing a small multilayer perceptron; the feature maps of the last layer go through global average pooling and connect to the output class nodes](/content/papers/assets/d94aa487d35fbfcdda30993d49c170efa0980e5c28d2a718b75659cd77164e3e.png)](/content/papers/assets/d94aa487d35fbfcdda30993d49c170efa0980e5c28d2a718b75659cd77164e3e.png)\n\n*Figure 2 from the paper, excerpted from [**page 4 of the v3 PDF**](https://arxiv.org/pdf/1312.4400v3#page=4). Every NiN in the paper's experiments has three mlpconv layers plus a global average pooling layer.*\n\nThe NiN used in the paper's experiments stacks three mlpconv layers, each containing a three-layer perceptron, and ends with global average pooling. Each mlpconv layer is followed by max pooling with stride 2; dropout is applied to the output of every mlpconv layer except the last, with weight decay on top. The authors put the detailed per-layer settings in the supplementary materials.\n\n## Discussion\n\nThe authors evaluate on four datasets: CIFAR-10, CIFAR-100, SVHN, and MNIST. Data preprocessing and the training/validation splits all follow the maxout paper; the implementation uses Alex Krizhevsky's cuda-convnet.\n\n### CIFAR-10 and CIFAR-100\n\nCIFAR-10 has 10 classes of $32\\times32$ color images, 50,000 for training and 10,000 for testing. The authors give each mlpconv layer the same number of feature maps as the maxout network, and tune only two hyperparameters on the validation set: the local receptive field size and weight decay.\n\n| Method | Test error |\n| --- | ---: |\n| Conv. maxout + Dropout | 11.68% |\n| **NIN + Dropout** | **10.41%** |\n| Conv. maxout + Dropout + data augmentation | 9.38% |\n| DropConnect + 12 networks + data augmentation | 9.32% |\n| **NIN + Dropout + data augmentation** | **8.81%** |\n\n*Table: Excerpted from Table 1 of the paper. Data augmentation means translation and horizontal flipping.*\n\nWithout data augmentation, NIN beats maxout by 1.27 percentage points; with translation and horizontal flipping, its 8.81% is also lower than DropConnect with 12 networks.\n\nDropout makes a real difference. The authors compare mlpconv layers with and without dropout between them: without dropout, test error is 14.51%; with it, error falls to 10.41%, which the authors describe as a relative reduction of more than 20%.\n\nCIFAR-100 has the same image format but 100 classes, with only a tenth as many images per class as CIFAR-10. The authors don't tune any hyperparameters for it; they just change the last layer to output 100 feature maps. Test error is 35.68%, more than one percentage point below the best result at the time without data augmentation (36.85%).\n\n### SVHN and MNIST\n\nSVHN is digit recognition on house numbers. NIN's error rate is 2.35%, lower than maxout's 2.47%. In the same table, though, Multi-digit Number Recognition gets 2.16% and DropConnect 1.94%, both better than NIN.\n\nOn MNIST, NIN gets 0.47% and maxout 0.45%. The authors' explanation is that MNIST has already been tuned to a very low error rate.\n\n### Does Global Average Pooling Really Regularize?\n\nTo look at GAP on its own, the authors swap it back for fully connected layers, leave everything else unchanged, and compare on CIFAR-10:\n\n| Top layers | Test error |\n| --- | ---: |\n| mlpconv + fully connected | 11.59% |\n| mlpconv + fully connected + Dropout | 10.88% |\n| mlpconv + global average pooling | 10.41% |\n\n*Table: Excerpted from Table 5 of the paper.*\n\nFully connected without dropout is the worst, adding dropout helps somewhat, and GAP is the best.\n\nThe authors then ask: does an ordinary convolutional network get the same effect from GAP? They try the network from Hinton et al.'s dropout paper (three convolutional layers plus one locally connected layer). With fully connected layers and no dropout it gets 17.56%, with dropout 15.99%, and with GAP 16.46%: about one percentage point better than no dropout, but worse than dropout. The authors' explanation is the precondition from earlier: asking linear convolutions plus ReLU to learn each class's confidence map directly may be asking too much.\n\n### Did the Last Layer Really Become Confidence Maps?\n\n[![Two 10-by-10 grids. Each row is one CIFAR-10 test image, with the original on the left followed by the last layer's feature maps for the 10 classes; the ones outlined in red along the diagonal are the feature maps for each image's true class, mostly the brightest, with the bright spots roughly where the object is in the original image](/content/papers/assets/c0032c1fba54d5052a18867e6bd6347143d3ba47c5cf4b353a61516107a0cf65.png)](/content/papers/assets/c0032c1fba54d5052a18867e6bd6347143d3ba47c5cf4b353a61516107a0cf65.png)\n\n*Figure 4 from the paper, excerpted from [**page 9 of the v3 PDF**](https://arxiv.org/pdf/1312.4400v3#page=9). Feature maps from the last mlpconv layer, showing only the top 10% of activations; the classes, in order, are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck, and the red boxes mark each image's true class. The left and right sets are simply different examples.*\n\nEach row is one test image and its response on the 10 feature maps. The red box marks the true-class map, which is usually the brightest in its row. GAP forces that by design anyway; the more interesting point is something else: on the true-class feature map, the brightest spots fall roughly where the object sits in the original image. The authors single out objects with clear structure, such as the car.\n\nThese feature maps were trained with class labels only and never saw a single bounding box. In their conclusion, the authors note that this makes object detection with NIN possible.\n\n## Conclusion\n\nNiN's two components were designed together: mlpconv gives each local patch a stronger model, and that is what allows GAP to demand that the last layer's feature maps correspond directly to classes. Experimentally, the CIFAR-10 and CIFAR-100 results and the GAP comparison both support this design; on an ordinary convolutional network, GAP used on its own loses some of its effect.\n\nNiN made its mark on ImageNet as well: according to the ILSVRC overview paper, NUS, winner of the 2014 detection challenge (in the track using only the provided data), used network-in-network within the R-CNN framework.\n\nBoth of NiN's components were later taken apart and used separately. $1\\times1$ convolutions adjust the channel count in the bottleneck structures of GoogLeNet and ResNet, and global average pooling handles the ending, though it's usually followed by one more fully connected layer and no longer requires each feature map to correspond to one class. The next time you open the code of a classification network, that `AdaptiveAvgPool2d(1)` line near the end traces back to this paper.\n\n\n> [!TIP]\n> \n> ResNet's bottleneck block uses $1\\times1$ convolutions to first reduce and then restore the channels, and the network ends with global average pooling followed by one fully connected layer:\n> \n> - **[[15.12] ResNet: Convolutional with 100 Layers](/blog/papers/classic-cnns/resnet)**\n\n\nAs for “the last layer's feature maps show where the object is,” that idea wasn't put to serious use until [CAM](/blog/papers/classic-cnns/cam); we'll pick it up again there.\n","imageSizes":{"/content/papers/assets/8024fc245eb852a2c0cd34b59c18c680b92766c33d621b934dbf102c1cacda13.png":{"width":896,"height":330},"/content/papers/assets/d94aa487d35fbfcdda30993d49c170efa0980e5c28d2a718b75659cd77164e3e.png":{"width":992,"height":296},"/content/papers/assets/c0032c1fba54d5052a18867e6bd6347143d3ba47c5cf4b353a61516107a0cf65.png":{"width":1078,"height":528}}}