{"notes":[{"id":"classic-cnns/cam","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"CAM","subtitle":"Where Is the Model Looking?","author":"Z. Yuan","paperDate":"2015-12-14","publishedOn":"2026-01-05","world":"external/cam","paperTitle":"Learning Deep Features for Discriminative Localization","paperUrl":"https://arxiv.org/abs/1512.04150","bibliography":{"title":"Learning Deep Features for Discriminative Localization","authors":["Bolei Zhou","Aditya Khosla","Agata Lapedriza","Aude Oliva","Antonio Torralba"],"url":"https://arxiv.org/abs/1512.04150","pdfUrl":"https://arxiv.org/pdf/1512.04150v1","codeUrl":null,"preprint":{"id":"1512.04150","submittedOn":"2015-12-14","url":"https://arxiv.org/abs/1512.04150"},"publication":null,"citation":"@misc{classiccnnscam2015,\n  title = {{Learning Deep Features for Discriminative Localization}},\n  author = {Bolei Zhou and Aditya Khosla and Agata Lapedriza and Aude Oliva and Antonio Torralba},\n  year = {2015},\n  eprint = {1512.04150},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.04150}\n}\n","citationPath":"/content/papers/citations/classic-cnns/cam.2ce837236f9d.bib"},"thumbnail":{"src":"/content/papers/assets/c3c6af7f1943634784331ca03a55efc3e68a7714d936d953a14f11b0828d9cf8.png","width":1372,"height":615},"locale":"en","path":"/content/papers/en/classic-cnns/cam.5e1bd71357c2.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"splitting-the-score-back-over-space","text":"Splitting the Score Back Over Space","depth":3},{"id":"average-vs-max-whats-the-difference","text":"Average vs. Max: What's the Difference?","depth":3},{"id":"raising-the-resolution-of-the-last-layer","text":"Raising the Resolution of the Last Layer","depth":3},{"id":"from-heat-map-to-box","text":"From Heat Map to Box","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-much-does-classification-suffer-without-the-fully-connected-layers","text":"How Much Does Classification Suffer Without the Fully Connected Layers?","depth":3},{"id":"weakly-supervised-localization","text":"Weakly Supervised Localization","depth":3},{"id":"moving-to-other-tasks","text":"Moving to Other Tasks","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/nin","relation":"referenced","topic":""},{"id":"classic-cnns/zfnet","relation":"referenced","topic":""},{"id":"classic-cnns/resnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/nin","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NiN","subtitle":"Foreshadowing the 1×1","author":"Z. Yuan","paperDate":"2013-12-16","publishedOn":"2025-09-24","world":"external/network-in-network","paperTitle":"Network In Network","paperUrl":"https://arxiv.org/abs/1312.4400","bibliography":{"title":"Network In Network","authors":["Min Lin","Qiang Chen","Shuicheng Yan"],"url":"https://arxiv.org/abs/1312.4400","pdfUrl":"https://arxiv.org/pdf/1312.4400v3","codeUrl":null,"preprint":{"id":"1312.4400","submittedOn":"2013-12-16","url":"https://arxiv.org/abs/1312.4400"},"publication":null,"citation":"@misc{classiccnnsnin2013,\n  title = {{Network In Network}},\n  author = {Min Lin and Qiang Chen and Shuicheng Yan},\n  year = {2013},\n  eprint = {1312.4400},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.4400}\n}\n","citationPath":"/content/papers/citations/classic-cnns/nin.db58639ac47c.bib"},"thumbnail":{"src":"/content/papers/assets/8024fc245eb852a2c0cd34b59c18c680b92766c33d621b934dbf102c1cacda13.png","width":896,"height":330},"locale":"en","path":"/content/papers/en/classic-cnns/nin.55401b294dee.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"running-a-small-mlp-on-every-patch","text":"Running a Small MLP on Every Patch","depth":3},{"id":"replacing-the-fully-connected-layers-with-an-average","text":"Replacing the Fully Connected Layers with an Average","depth":3},{"id":"the-whole-network","text":"The Whole Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"cifar-10-and-cifar-100","text":"CIFAR-10 and CIFAR-100","depth":3},{"id":"svhn-and-mnist","text":"SVHN and MNIST","depth":3},{"id":"does-global-average-pooling-really-regularize","text":"Does Global Average Pooling Really Regularize?","depth":3},{"id":"did-the-last-layer-really-become-confidence-maps","text":"Did the Last Layer Really Become Confidence Maps?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/cam","relation":"referenced","topic":""}]},{"id":"classic-cnns/zfnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ZFNet","subtitle":"The View Inside the Black Box","author":"Z. Yuan","paperDate":"2013-11-12","publishedOn":"2025-09-06","world":"external/zfnet","paperTitle":"Visualizing and Understanding Convolutional Networks","paperUrl":"https://arxiv.org/abs/1311.2901","bibliography":{"title":"Visualizing and Understanding Convolutional Networks","authors":["Matthew D Zeiler","Rob Fergus"],"url":"https://arxiv.org/abs/1311.2901","pdfUrl":"https://arxiv.org/pdf/1311.2901v3","codeUrl":null,"preprint":{"id":"1311.2901","submittedOn":"2013-11-12","url":"https://arxiv.org/abs/1311.2901"},"publication":null,"citation":"@misc{classiccnnszfnet2013,\n  title = {{Visualizing and Understanding Convolutional Networks}},\n  author = {Matthew D Zeiler and Rob Fergus},\n  year = {2013},\n  eprint = {1311.2901},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1311.2901}\n}\n","citationPath":"/content/papers/citations/classic-cnns/zfnet.6d4dd35e9f91.bib"},"thumbnail":{"src":"/content/papers/assets/6c45d745e8a2f4b60cc45bea1029353e303d5e3bbee1e62f438c71874fddf883.png","width":1200,"height":1329},"locale":"en","path":"/content/papers/en/classic-cnns/zfnet.a0c5a08699b9.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"walking-backward-through-a-convolutional-network","text":"Walking Backward Through a Convolutional Network","depth":3},{"id":"unpooling-has-to-remember-where-the-max-was","text":"Unpooling Has to Remember Where the Max Was","depth":3},{"id":"the-model-under-observation","text":"The Model Under Observation","depth":3},{"id":"what-each-layer-sees","text":"What Each Layer Sees","depth":3},{"id":"fixing-the-architecture-with-what-you-see","text":"Fixing the Architecture with What You See","depth":3},{"id":"cover-a-patch-and-watch-the-classifier","text":"Cover a Patch and Watch the Classifier","depth":3},{"id":"correspondence-across-images","text":"Correspondence Across Images","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet-2012","text":"ImageNet 2012","depth":3},{"id":"which-layers-hurt-most-when-removed","text":"Which Layers Hurt Most When Removed?","depth":3},{"id":"switching-datasets","text":"Switching Datasets","depth":3},{"id":"are-higher-layers-more-useful","text":"Are Higher Layers More Useful?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/alexnet","relation":"referenced","topic":""},{"id":"classic-cnns/vgg","relation":"referenced","topic":""},{"id":"classic-cnns/resnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"Convolutional with 100 Layers","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"en","path":"/content/papers/en/classic-cnns/resnet.b65375cb9846.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-is-residual","text":"What is Residual?","depth":3},{"id":"residual-learning","text":"Residual Learning","depth":3},{"id":"residual-module","text":"Residual Module","depth":3},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"architectures-and-computation","text":"Architectures and Computation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"deeper-bottleneck-architectures","text":"Deeper Bottleneck Architectures","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"Training deeper convolutional networks"},{"id":"classic-cnns/densenet","relation":"next","topic":"Training deeper convolutional networks"}]}],"text":"Does a classification network that has only ever seen class labels know where the object is?\n\nIn work from the year before, Bolei Zhou and colleagues at MIT found that the answer is mostly yes: in a convolutional network trained for scene classification, the convolutional units in various layers behave much like object detectors, even though nobody told the network where the objects were during training. But this ability disappears at the fully connected layers. The output of a convolutional layer still keeps spatial positions; once it's flattened and fed into fully connected layers, “where” and “what” get mixed together and can no longer be pulled apart.\n\nIn December 2015, the same group (Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba) released this paper, later published at CVPR 2016. They turned their attention to a component that already existed: global average pooling, proposed in NiN and also used in GoogLeNet.\n\n\n> [!TIP]\n> \n> For where global average pooling came from, and what NiN's last-layer feature maps look like, you can start with our note:\n> \n> - **[[13.12] NiN: Foreshadowing the 1×1](/blog/papers/classic-cnns/nin)**\n\n\n## Defining the Problem\n\nThis paper tackles **weakly supervised object localization**: training has only image-level class labels, yet at test time the model has to put a box around the object.\n\nSeveral approaches already existed. Some covered up the image piece by piece to see where occlusion made the response drop most; some combined multiple-instance learning with convolutional features; some ran the network once on each of a large number of overlapping patches. None of these were trained end to end, and they required many forward passes, which made them hard to use on large-scale data.\n\nThe closest was the method of Oquab et al., which localizes objects with global **max** pooling. It can only find a single point on the object's boundary and can't tell you how far the object extends.\n\nAnother related line of work is visualizing convolutional networks. [ZFNet](/blog/papers/classic-cnns/zfnet) used a deconvnet to see what pattern each unit looks for, and Zhou et al.'s own earlier work found object detectors inside scene networks; all of this research analyzed only the convolutional layers and skipped the fully connected ones.\n\nWhat the authors wanted: with a single forward pass, the network can tell you which regions of the image it used to decide on this class.\n\n## Solving the Problem\n\n### Splitting the Score Back Over Space\n\nThe authors use a network structure similar to NiN and GoogLeNet: mostly convolutional layers up front, global average pooling after the last convolutional layer, then one fully connected layer, then a softmax.\n\nLet $f_k(x, y)$ be the activation of the $k$-th unit of the last convolutional layer at position $(x, y)$. Global average pooling squeezes each feature map into a single number:\n\n$$\nF_k = \\sum_{x,y} f_k(x, y)\n$$\n\nThe paper writes this as a plain sum and leaves out the division by the number of positions; the two differ by a constant, which doesn't affect the derivation that follows.\n\nThe score for class $c$ is a weighted sum of these numbers:\n\n$$\nS_c = \\sum_k w_k^c F_k\n$$\n\n$w_k^c$ is a weight of the fully connected layer, representing how important the $k$-th feature map is for class $c$. The authors set the softmax bias to 0, saying it has almost no effect on classification.\n\nNext comes the key to the whole paper. Substitute $F_k$ and swap the order of the two sums:\n\n$$\nS_c = \\sum_k w_k^c \\sum_{x,y} f_k(x, y) = \\sum_{x,y} \\sum_k w_k^c f_k(x, y)\n$$\n\nThe inner sum $\\sum_k w_k^c f_k(x, y)$ still carries the position $(x, y)$. The authors define it as the **class activation map** (CAM) for class $c$:\n\n$$\nM_c(x, y) = \\sum_k w_k^c f_k(x, y)\n$$\n\nSo $S_c = \\sum_{x,y} M_c(x, y)$. The class score is simply the sum of the CAM over all positions, and $M_c(x, y)$ tells you directly how much position $(x, y)$ contributes to that score.\n\nIn other words, classification averages first and weights second; CAM reverses the order, weighting first and keeping the spatial layout instead of averaging. Let's check with a small example:\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nK, C = 4, 3                                   # 4 feature maps, 3 classes\nf = torch.relu(torch.randn(K, 7, 7))          # output of the last conv layer: 4 × 7 × 7\nw = torch.randn(C, K)                         # the fully connected layer after GAP: 4 weights per class\n\n# Classification: average first, then weight\nscores = w @ f.mean(dim=(1, 2))               # [3]\n\n# CAM: weight first, keep the spatial layout\ncams = torch.einsum('ck,khw->chw', w, f)      # [3, 7, 7]\n\nprint(\"Class scores:    \", scores)\nprint(\"Mean of each CAM:\", cams.mean(dim=(1, 2)))\nprint(cams.shape)\n```\n\n```text\nClass scores:     tensor([-1.0399, -0.8521, -0.5210])\nMean of each CAM: tensor([-1.0399, -0.8521, -0.5210])\ntorch.Size([3, 7, 7])\n```\n\nThe two sets of numbers are identical. CAM uses no new parameters and needs no second pass through the network; it just “projects” the fully connected weights the network has already learned back onto the feature maps of the last convolutional layer.\n\n[![The top half is the network: a photo of a boy holding a dog passes through a series of convolutional layers, and the feature maps of the last convolutional layer go through GAP and connect to the class Australian terrier with weights w1, w2, through wn. The bottom half is titled Class Activation Mapping: w1, w2, and wn each multiply their corresponding feature map, and the results are summed into a heat map overlaid on the original image, brightest over the dog](/content/papers/assets/c3c6af7f1943634784331ca03a55efc3e68a7714d936d953a14f11b0828d9cf8.png)](/content/papers/assets/c3c6af7f1943634784331ca03a55efc3e68a7714d936d953a14f11b0828d9cf8.png)\n\n*Figure 2 from the paper, excerpted from [**page 3 of the v1 PDF**](https://arxiv.org/pdf/1512.04150v1#page=3). The predicted class score is mapped back to the previous convolutional layer to produce the class activation map.*\n\nThe bottom half of the figure is the computation of $M_c$. Each feature map detects some visual pattern of its own: some light up on the dog's head, others on the body below. Multiply them by the weights for the class “Australian terrier” and add them up, and you get this class's heat map. The last convolutional layer has very low resolution ($13\\times13$ or $14\\times14$ in the paper); upsample the CAM to the size of the input image, and you can see which regions of the image are most relevant to this class.\n\nFor the same image, a different class means a different set of weights, and the CAM changes with it:\n\n[![Top left is a nighttime photo of a domed building, labeled dome. The other five panels show the CAMs and scores for the top five predicted classes: palace 0.459 lights up on the wide palace building below; dome 0.195 on the dome above; church 0.146, altar 0.091, and monastery 0.051 concentrate their bright spots around the lower left of the building and the spire](/content/papers/assets/5feae5d21639fe423e9457b6eee11d7c4357e0fab4014dac2dd2482a810f0c45.png)](/content/papers/assets/5feae5d21639fe423e9457b6eee11d7c4357e0fab4014dac2dd2482a810f0c45.png)\n\n*Figure 4 from the paper, excerpted from [**page 4 of the v1 PDF**](https://arxiv.org/pdf/1512.04150v1#page=4). The true class is dome; the other five panels are the CAMs of the top five predictions, each labeled at the top with its class and score.*\n\nPalace lights up on the flat main building below, and dome on the dome above. To judge the two classes, the network looks at different parts of the same image.\n\n### Average vs. Max: What's the Difference?\n\nSince Oquab et al. used global max pooling, the authors take care to explain why they use the average.\n\nTheir intuition: with an average, the best way to raise the score is to find **all** the discriminative parts of the object, since a low activation anywhere drags the average down; with a maximum, only the strongest position counts, and whether the other positions are high or low makes no difference.\n\nThe difference shows up most clearly in backpropagation:\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nx = torch.rand(1, 1, 4, 4, requires_grad=True)\n\nx.mean().backward()                           # global average pooling\nprint(\"GAP gradient:\\n\", x.grad[0, 0])\n\nx.grad = None\nx.amax().backward()                           # global max pooling\nprint(\"GMP gradient:\\n\", x.grad[0, 0])\n```\n\n```text\nGAP gradient:\n tensor([[0.0625, 0.0625, 0.0625, 0.0625],\n        [0.0625, 0.0625, 0.0625, 0.0625],\n        [0.0625, 0.0625, 0.0625, 0.0625],\n        [0.0625, 0.0625, 0.0625, 0.0625]])\nGMP gradient:\n tensor([[0., 0., 0., 0.],\n        [0., 0., 0., 1.],\n        [0., 0., 0., 0.],\n        [0., 0., 0., 0.]])\n```\n\nAverage pooling splits the gradient evenly over all 16 positions, so every position gets trained; max pooling sends the gradient only to the largest cell, and the other 15 learn nothing from this step. The experiments later check whether this intuition holds up for localization.\n\n### Raising the Resolution of the Last Layer\n\nThe authors modify three networks, AlexNet, VGGnet, and GoogLeNet: they remove the final fully connected layers and replace them with GAP followed by a fully connected softmax layer.\n\nThey found that the higher the spatial resolution of the last convolutional layer before GAP, the better the localization. The authors call this resolution the mapping resolution. To raise it, they also removed a few more layers:\n\n- **AlexNet**: remove all layers after conv5, giving $13\\times13$.\n- **VGGnet**: remove all layers after conv5-3, giving $14\\times14$.\n- **GoogLeNet**: remove all layers after inception4e, giving $14\\times14$.\n\nEach network then gets one more convolutional layer, $3\\times3$ with stride 1, padding 1, and 1024 units, followed by GAP and softmax, and is fine-tuned for 1000-class classification on ILSVRC's 1.3 million training images. The resulting networks are called AlexNet-GAP, VGGnet-GAP, and GoogLeNet-GAP. The authors mention in a footnote that training from scratch gives similar results.\n\n### From Heat Map to Box\n\nILSVRC's localization evaluation wants a box. The authors' method is simple:\n\n1. Upsample the CAM to the image size.\n2. Keep the regions whose values exceed 20% of the maximum.\n3. Take the largest connected region and draw its bounding box.\n\nLet's walk through it with a fake $14\\times14$ CAM that has a large response on the left and a small one in the top-right corner:\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom scipy import ndimage\n\n# A 14×14 CAM: a large response on the left, a small one in the top-right corner\nyy, xx = torch.meshgrid(torch.arange(14.), torch.arange(14.), indexing='ij')\ncam = 1.0 * torch.exp(-((yy - 8) ** 2 / 6 + (xx - 4) ** 2 / 4)) \\\n    + 0.6 * torch.exp(-((yy - 2) ** 2 + (xx - 11) ** 2) / 1.5)\n\n# 1. Upsample back to the input image size\ncam = F.interpolate(cam[None, None], size=(224, 224), mode='bilinear', align_corners=False)[0, 0]\n\n# 2. Keep the regions above 20% of the maximum\nmask = (cam > 0.2 * cam.max()).numpy()\n\n# 3. Find the largest connected region and take its bounding box\nlabels, n = ndimage.label(mask)\nsizes = ndimage.sum(mask, labels, range(1, n + 1))\nlargest = labels == (sizes.argmax() + 1)\nrows, cols = largest.any(axis=1).nonzero()[0], largest.any(axis=0).nonzero()[0]\n\nprint(\"Connected regions:\", n, \"with pixel counts:\", sizes.astype(int).tolist())\nprint(\"Box (x0, y0, x1, y1):\", cols[0], rows[0], cols[-1], rows[-1])\n```\n\n```text\nConnected regions: 2 with pixel counts: [1344, 6552]\nBox (x0, y0, x1, y1): 30 85 113 186\n```\n\nThe top-right region also passes the threshold, but it's small, so the final box encloses only the large region on the left. This also shows a limitation of the method: when an image contains two objects, it boxes only one of them. For top-5 localization evaluation, the procedure is run once for each of the top five predicted classes.\n\n## Discussion\n\n### How Much Does Classification Suffer Without the Fully Connected Layers?\n\n| Network | Top-1 error | Top-5 error |\n| --- | ---: | ---: |\n| AlexNet | 42.6 | 19.5 |\n| AlexNet-GAP | 51.1 | 26.3 |\n| AlexNet\\*-GAP | 44.9 | 20.9 |\n| VGGnet | 31.2 | 11.4 |\n| VGGnet-GAP | 33.4 | 12.2 |\n| GoogLeNet | 31.9 | 11.3 |\n| GoogLeNet-GAP | 35.0 | 13.2 |\n| GoogLeNet-GMP | 35.6 | 13.9 |\n| NIN | 41.9 | 19.6 |\n\n*Table: Excerpted from Table 1 of the paper, ILSVRC validation set, in percent; the rows have been reordered for easier comparison with the original networks.*\n\nAfter VGGnet and GoogLeNet lose a few layers, their top-1 error rises by 2.2 and 3.1 percentage points, respectively, and top-5 by less than 2 points for both. AlexNet is hit hardest, with top-1 error going from 42.6% to 51.1%; when the authors add two more convolutional layers before GAP, the resulting AlexNet\\*-GAP is about as good as the original AlexNet. GAP and GMP also perform similarly on classification, as expected.\n\nClassification can't afford to get much worse, because the localization evaluation requires both the class and the box to be right.\n\n### Weakly Supervised Localization\n\n| Method | Top-1 error | Top-5 error |\n| --- | ---: | ---: |\n| **GoogLeNet-GAP** | **56.40** | **43.00** |\n| VGGnet-GAP | 57.20 | 45.14 |\n| GoogLeNet-GMP | 57.78 | 45.26 |\n| GoogLeNet | 60.09 | 49.34 |\n| AlexNet\\*-GAP | 63.75 | 49.53 |\n| Backprop on GoogLeNet | 61.31 | 50.55 |\n| Backprop on VGGnet | 61.12 | 51.46 |\n| AlexNet-GAP | 67.19 | 52.16 |\n| Backprop on AlexNet | 65.17 | 52.64 |\n| NIN | 65.47 | 54.19 |\n\n*Table: Excerpted from Table 2 of the paper: localization error on the ILSVRC validation set, sorted by top-5. Backprop means using the backpropagation saliency maps of Simonyan et al. instead of CAM.*\n\nA few comparisons:\n\n- **CAM vs. backpropagation**: on GoogLeNet and VGGnet, CAM's top-5 error is more than 6 percentage points lower than backpropagation's; on AlexNet the two are close (52.16 vs. 52.64), and on top-1 backpropagation actually comes out slightly ahead.\n- **GoogLeNet-GAP vs. GoogLeNet**: the original GoogLeNet classifies better but localizes worse than the trimmed GoogLeNet-GAP. The authors attribute this to the original GoogLeNet's last layer being only $7\\times7$, too low a resolution.\n- **GAP vs. GMP**: similar on classification, but for localization GAP's top-5 error is 2.26 percentage points lower, supporting the earlier intuition about average pooling.\n\n[![Two groups, left and right. a) Four examples, each with an original image and a CAM: American alligator, maze, Shetland sheepdog, and grasshopper; on the original images, green boxes are the annotations and red boxes are inferred from the CAM, and the two roughly overlap. b) The top two examples are GoogLeNet-GAP localizing a snowmobile and an apron, with the heat concentrated on the object; the bottom two are results of backpropagation on AlexNet, where the heat map is scattered dots and the inferred boxes are too large](/content/papers/assets/dce42d560109e416d7229359699d2bca63f7ec9a74374b2d62fce387c9bbed22.png)](/content/papers/assets/dce42d560109e416d7229359699d2bca63f7ec9a74374b2d62fce387c9bbed22.png)\n\n*Figure 6 from the paper, excerpted from [**page 6 of the v1 PDF**](https://arxiv.org/pdf/1512.04150v1#page=6). a) Localization examples from GoogLeNet-GAP; b) the top two examples are GoogLeNet-GAP and the bottom two are backpropagation on AlexNet. Green boxes are annotations; red boxes are inferred from the heat maps.*\n\nThe comparison on the right is easy to read: the backpropagation saliency map is a spread of scattered bright dots, so after thresholding the box comes out too large; CAM's hot region is concentrated, and its box fits the object more closely.\n\nOn the test set, the authors switch to a different box-selection strategy: for each of the top two classes, take one tight box and one loose box, and for the third class, one loose box. This heuristic was tuned on the validation set.\n\n| Method | Supervision | Top-5 test error |\n| --- | --- | ---: |\n| GoogLeNet-GAP (heuristic box selection) | Weak | **37.1** |\n| GoogLeNet-GAP | Weak | 42.9 |\n| Backprop | Weak | 46.4 |\n| GoogLeNet | Full | 26.7 |\n| OverFeat | Full | 29.9 |\n| AlexNet | Full | 34.2 |\n\n*Table: Excerpted from Table 3 of the paper, ILSVRC test set.*\n\n37.1% is already close to the fully supervised AlexNet (34.2%), and GoogLeNet-GAP never saw a single box during training. The authors also acknowledge that the gap is still large: with the same GoogLeNet architecture, full supervision reaches 26.7%.\n\n### Moving to Other Tasks\n\nThe authors also wanted to know whether these localization-capable features could be used directly on other datasets. Their approach: freeze GoogLeNet-GAP, train a linear SVM on the GAP output, and compute the CAM with the SVM weights in place of the original softmax weights.\n\nAs generic image features, GoogLeNet-GAP performs about the same as the features from the original GoogLeNet's average pooling layer on eight datasets, including SUN397, MIT Indoor67, and Caltech256, and is mostly well ahead of AlexNet's fc7.\n\nThe fine-grained CUB-200-2011 bird dataset is more interesting. It has 200 bird species, 5,994 training images, and 5,794 test images, and it comes with bounding boxes for the birds.\n\n| Method | Annotations used in training/testing | Accuracy |\n| --- | --- | ---: |\n| GoogLeNet-GAP, whole image | None | 63.0% |\n| GoogLeNet-GAP, boxes inferred from CAM | None | 67.8% |\n| GoogLeNet-GAP, annotated boxes | Boxes | 70.5% |\n| PANDA R-CNN | Boxes + parts | 76.4% |\n\n*Table: Selected rows excerpted from Table 4 of the paper.*\n\nBoxing the bird with CAM first and then re-extracting features from inside the box raises accuracy from 63.0% to 67.8%, without any annotation. Under the IoU 0.5 criterion, 41.0% of the boxes inferred from CAM are correct, against 5.5% for random guessing. Fine-grained differences often come down to a few details, so focusing on the bird first clearly helps.\n\nThe last set of experiments is about “pattern discovery”: given a group of images that share a concept, does the CAM light up on that concept? One of the examples concerns text detection:\n\n[![The top row has six street-view photos of storefronts, and the bottom row shows the corresponding CAMs. The hot regions all fall on the text of the shop signs, such as store names on awnings and signs above doorways](/content/papers/assets/89c6a4410389b611bdcc7e40a414223d52106f89494e17ede86188b4552d73a0.png)](/content/papers/assets/89c6a4410389b611bdcc7e40a414223d52106f89494e17ede86188b4552d73a0.png)\n\n*Figure 11 from the paper, excerpted from [**page 8 of the v1 PDF**](https://arxiv.org/pdf/1512.04150v1#page=8). A weakly supervised text detector: the network wasn't trained with text or any boxes, yet the CAM still lights up on the text.*\n\nThe positive set is 350 Google Street View images containing text, from the SVT dataset, and the negative set is outdoor scenes randomly sampled from the SUN dataset. With nothing but a binary label for “does this image contain text,” the CAM lands on the text of the signs. The paper also tries localizing informative objects in scenes and concepts described by short phrases, and interpreting the answers of visual question answering models.\n\nThe authors did one more thing: using the weights after GAP, they ranked the units of the last convolutional layer by their importance for a given class, which shows directly which units matter most when judging that class. For example, the units important for the lakeland terrier (a dog breed) detect dog faces and body fur; the ones important for living room detect sofas, tables, and fireplaces.\n\n## Conclusion\n\nCAM itself is a one-line formula: multiply the classification layer's weights back onto the last layer's feature maps. It works because GAP is followed by only one linear layer, so the class score can be split back over every position. That's also its precondition: the network must have a “convolutions, GAP, one fully connected layer” structure. Other networks have to be re-architected and fine-tuned first, and their classification accuracy may drop as a result.\n\nExperimentally, CAM reaches 37.1% top-5 error on ILSVRC weakly supervised localization, close to the fully supervised AlexNet but still some distance from the fully supervised GoogLeNet with the same architecture. Taking only the box of the largest connected region per image also makes it better suited to images with one main object.\n\nCAM's architectural constraint was soon addressed: [Grad-CAM](https://arxiv.org/abs/1610.02391), from October 2016, uses gradients to set each feature map's weight, with no need to switch to a GAP architecture or retrain.\n\nCAM has a low barrier to entry: as long as the network ends in GAP followed by one fully connected layer, a few lines of code will compute it. The now-common [ResNet](/blog/papers/classic-cnns/resnet) has exactly this structure, so if you want to know where your model is looking, you can draw one in no time.\n\nJust don't take the heat map too seriously. It marks the most discriminative parts, such as an animal's head, and often covers only part of the object.\n","imageSizes":{"/content/papers/assets/c3c6af7f1943634784331ca03a55efc3e68a7714d936d953a14f11b0828d9cf8.png":{"width":1372,"height":615},"/content/papers/assets/5feae5d21639fe423e9457b6eee11d7c4357e0fab4014dac2dd2482a810f0c45.png":{"width":672,"height":462},"/content/papers/assets/dce42d560109e416d7229359699d2bca63f7ec9a74374b2d62fce387c9bbed22.png":{"width":1381,"height":304},"/content/papers/assets/89c6a4410389b611bdcc7e40a414223d52106f89494e17ede86188b4552d73a0.png":{"width":669,"height":235}}}