{"notes":[{"id":"classic-cnns/xception","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Xception","subtitle":"Where Inception Ends","author":"Z. Yuan","paperDate":"2016-10-07","publishedOn":"2024-03-20","world":"external/xception","paperTitle":"Xception: Deep Learning with Depthwise Separable Convolutions","paperUrl":"https://arxiv.org/abs/1610.02357","bibliography":{"title":"Xception: Deep Learning with Depthwise Separable Convolutions","authors":["François Chollet"],"url":"https://arxiv.org/abs/1610.02357","pdfUrl":"https://arxiv.org/pdf/1610.02357v3","codeUrl":null,"preprint":{"id":"1610.02357","submittedOn":"2016-10-07","url":"https://arxiv.org/abs/1610.02357"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2017/html/Chollet_Xception_Deep_Learning_CVPR_2017_paper.html"},"citation":"@misc{classiccnnsxception2016,\n  title = {{Xception: Deep Learning with Depthwise Separable Convolutions}},\n  author = {François Chollet},\n  year = {2016},\n  eprint = {1610.02357},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1610.02357}\n}\n","citationPath":"/content/papers/citations/classic-cnns/xception.3a6fa7188c36.bib"},"thumbnail":{"src":"/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png","width":917,"height":538},"locale":"en","path":"/content/papers/en/classic-cnns/xception.b83ea41802e4.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"inception-from-another-angle","text":"Inception from Another Angle","depth":3},{"id":"splitting-all-the-way","text":"Splitting All the Way","depth":3},{"id":"the-other-end-of-the-spectrum","text":"The Other End of the Spectrum","depth":3},{"id":"xception","text":"Xception","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"jft","text":"JFT","depth":3},{"id":"similar-parameters-a-bit-slower","text":"Similar Parameters, a Bit Slower","depth":3},{"id":"residual-connections","text":"Residual Connections","depth":3},{"id":"an-activation-in-between","text":"An Activation in Between?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"lightweight/mobilenet-v1","relation":"referenced","topic":""}]},{"id":"classic-cnns/inception-v1","category":"classic-cnns","legacyPath":"classic-cnns/1409-inception-v1","legacySlug":"1409-inception-v1","legacyPaths":[],"title":"GoogLeNet","subtitle":"Also Known as Inception-V1","author":"Z. Yuan","paperDate":"2014-09-17","publishedOn":"2024-09-22","paperTitle":"Going Deeper with Convolutions","paperUrl":"https://arxiv.org/abs/1409.4842","bibliography":{"title":"Going Deeper with Convolutions","authors":["Christian Szegedy","Wei Liu","Yangqing Jia","Pierre Sermanet","Scott Reed","Dragomir Anguelov","Dumitru Erhan","Vincent Vanhoucke","Andrew Rabinovich"],"url":"https://arxiv.org/abs/1409.4842","pdfUrl":"https://arxiv.org/pdf/1409.4842","codeUrl":null,"preprint":{"id":"1409.4842","submittedOn":"2014-09-17","url":"https://arxiv.org/abs/1409.4842"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2015.7298594"},"citation":"@misc{classiccnnsinceptionv12014,\n  title = {{Going Deeper with Convolutions}},\n  author = {Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich},\n  year = {2014},\n  eprint = {1409.4842},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1409.4842}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v1.3d20e0e32bcd.bib"},"thumbnail":{"src":"/content/papers/assets/254b970b1374c24267b60c2d684338760e9ba8b10dad761a5fa15f1bb8bcf6ef.jpg","width":1330,"height":394},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v1.41dca1189319.md","headings":[{"id":"problem-solved","text":"Problem Solved","depth":2},{"id":"the-inception-module","text":"The Inception Module","depth":3},{"id":"configuration-parameters","text":"Configuration Parameters","depth":3},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"performance-at-ilsvrc-2014","text":"Performance at ILSVRC 2014","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnext","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"Big Kernels Bow Out","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2023-11-15","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html"},"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v3.e66992f80aec.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"splitting-55-into-two-33-layers","text":"Splitting 5×5 into Two 3×3 Layers","depth":3},{"id":"splitting-further-into-1n-and-n1","text":"Splitting Further into 1×n and n×1","depth":3},{"id":"what-auxiliary-classifiers-really-do","text":"What Auxiliary Classifiers Really Do","depth":3},{"id":"shrinking-the-feature-map-without-choking-the-information","text":"Shrinking the Feature Map Without Choking the Information","depth":3},{"id":"putting-the-pieces-together","text":"Putting the Pieces Together","depth":3},{"id":"keeping-the-model-from-getting-too-sure-of-itself","text":"Keeping the Model from Getting Too Sure of Itself","depth":3},{"id":"training-setup","text":"Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"adding-one-change-at-a-time","text":"Adding One Change at a Time","depth":3},{"id":"how-much-does-lower-resolution-cost","text":"How Much Does Lower Resolution Cost?","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]},{"id":"lightweight/mobilenet-v1","category":"lightweight","legacyPath":"lightweight/1704-mobilenet-v1","legacySlug":"1704-mobilenet-v1","legacyPaths":[],"title":"MobileNet-V1","subtitle":"Pioneers of Depthwise Separable Convolutions","author":"Z. Yuan","paperDate":"2017-04-17","publishedOn":"2024-06-25","paperTitle":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","paperUrl":"https://arxiv.org/abs/1704.04861","bibliography":{"title":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","authors":["Andrew G. Howard","Menglong Zhu","Bo Chen","Dmitry Kalenichenko","Weijun Wang","Tobias Weyand","Marco Andreetto","Hartwig Adam"],"url":"https://arxiv.org/abs/1704.04861","pdfUrl":"https://arxiv.org/pdf/1704.04861","codeUrl":null,"preprint":{"id":"1704.04861","submittedOn":"2017-04-17","url":"https://arxiv.org/abs/1704.04861"},"publication":null,"citation":"@misc{lightweightmobilenetv12017,\n  title = {{MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications}},\n  author = {Andrew G. Howard and Menglong Zhu and Bo Chen and Dmitry Kalenichenko and Weijun Wang and Tobias Weyand and Marco Andreetto and Hartwig Adam},\n  year = {2017},\n  eprint = {1704.04861},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1704.04861}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v1.e9c4c525fed4.bib"},"thumbnail":{"src":"/content/papers/assets/b8390bd8e0f21f1b1eaa6b2a5d4a1b0d9eba36e2fe5dd0878830d757839d30f1.jpg","width":1064,"height":1080},"locale":"en","path":"/content/papers/en/lightweight/mobilenet-v1.602fe825c0c0.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"depthwise-separable-convolution","text":"Depthwise Separable Convolution","depth":3},{"id":"network-architecture","text":"Network Architecture","depth":2},{"id":"training-techniques","text":"Training Techniques","depth":2},{"id":"discussion","text":"Discussion","depth":2},{"id":"cost-of-depthwise-separable-convolution","text":"Cost of Depthwise Separable Convolution","depth":3},{"id":"depth-or-width","text":"Depth or Width?","depth":3},{"id":"comparing-other-models","text":"Comparing Other Models","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v2","relation":"next","topic":"Efficient convolution on mobile devices"}]}],"text":"By 2016, Inception was already in its fourth generation, from GoogLeNet, through Inception-v2 with BN added and Inception-v3 with factorized convolutions, all the way to Inception-ResNet with residuals wired in. Every generation ranked near the top on ImageNet, but the modules grew more complicated each time, with branches, channel counts, and kernel shapes all needing to be tuned one by one.\n\n\n> [!TIP]\n> \n> For the Inception series, you can start with these two:\n> \n> - **[[14.09] GoogLeNet: Also Known as Inception-V1](/blog/papers/classic-cnns/inception-v1)**\n> - **[[15.12] Inception-v3: Big Kernels Bow Out](/blog/papers/classic-cnns/inception-v3)**\n\n\nFrançois Chollet, the creator of Keras, was at Google at the time. He wanted to understand why Inception modules work, how they differ from ordinary convolutions, and where things could go after Inception.\n\nThis paper is his answer.\n\n## Defining the Problem\n\nIn an ordinary convolutional layer, each kernel is a 3D filter, with a channel dimension on top of width and height. A $3\\times3$ kernel looking at 256 input channels has $3\\times3\\times256$ weights, and it handles **at the same time** the relationships between different channels at the same position (cross-channel correlations) and the relationships between neighboring positions (spatial correlations).\n\nThe Inception module handles these two things separately. It first uses a few $1\\times1$ convolutions to deal with cross-channel relationships, projecting the input into 3 or 4 smaller spaces, then uses $3\\times3$ or $5\\times5$ convolutions within each smaller space to deal with spatial relationships.\n\nThe author calls the idea behind this design the **Inception hypothesis**, namely that cross-channel correlations and spatial correlations are independent enough that handling them separately works better than handling them together.\n\nIf “partly separating them” works, what about separating them more thoroughly?\n\n## Solving the Problem\n\n### Inception from Another Angle\n\nThe author first simplifies the Inception module, keeping only one kind of convolution, $3\\times3$, and removing the average-pooling branch. That leaves three branches, each a $1\\times1$ followed by a $3\\times3$.\n\nThese three $1\\times1$ convolutions all look at the same input, so they can be merged into a single $1\\times1$ convolution with more output channels. After the merge, each of the three $3\\times3$ convolutions handles its own segment of those output channels:\n\n[![The equivalent form of the simplified Inception module, from bottom to top: Input goes into one 1x1 conv, whose Output channels are split by braces into three segments, each feeding its own 3x3 conv, and finally Concat](/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png)](/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png)\n\n*Figure 3 from the paper, excerpted from [**page 2 of the v3 PDF**](https://arxiv.org/pdf/1610.02357v3#page=2). A strictly equivalent reformulation of the simplified Inception module.*\n\nAre the two forms really the same? Let's try it on a 64-channel input. Each of the three branches does a $1\\times1$ (64 to 32) and a $3\\times3$; on the other side, a 64-to-96 $1\\times1$ convolution is followed by a convolution that “splits the 96 channels into three groups and runs a $3\\times3$ within each group,” with the weights copied straight over from the three branches:\n\n```python\nimport torch\nimport torch.nn as nn\n\ntorch.manual_seed(0)\nx = torch.randn(1, 64, 8, 8)                       # a 64-channel input\n\n# Figure 2: three branches, each a 1×1 followed by a 3×3\npw = [nn.Conv2d(64, 32, 1, bias=False) for _ in range(3)]\nsp = [nn.Conv2d(32, 32, 3, padding=1, bias=False) for _ in range(3)]\ny2 = torch.cat([s(p(x)) for p, s in zip(pw, sp)], dim=1)\n\n# Figure 3: one large 1×1 convolution, then the 96 output channels cut into three segments, each with its own 3×3\nbig_pw = nn.Conv2d(64, 96, 1, bias=False)\ngrouped = nn.Conv2d(96, 96, 3, padding=1, groups=3, bias=False)\nwith torch.no_grad():\n    big_pw.weight.copy_(torch.cat([p.weight for p in pw]))\n    grouped.weight.copy_(torch.cat([s.weight for s in sp]))\ny3 = grouped(big_pw(x))\n\nprint(\"Output shapes:\", tuple(y2.shape), tuple(y3.shape))\nprint(\"Max difference between the two forms:\", (y2 - y3).abs().max().item())\n```\n\n```text\nOutput shapes: (1, 96, 8, 8) (1, 96, 8, 8)\nMax difference between the two forms: 7.748603820800781e-07\n```\n\nThe difference is just floating-point rounding error. PyTorch's `groups=3` means exactly “split the channels into three segments and convolve each one separately,” so this simplified Inception module is really just a $1\\times1$ convolution plus a $3\\times3$ convolution split into three groups.\n\nSeen this way, the simplified Inception module has only one number left to tune, the number of segments the channels are split into.\n\n### Splitting All the Way\n\nThe more segments, the fewer channels in each. Taken to the extreme, each segment has just one channel, and every output channel gets a $3\\times3$ convolution of its own:\n\n[![The extreme version of the Inception module, from bottom to top: Input goes into one 1x1 conv, the output channels are split into many segments, each feeding a single 3x3, and finally Concat](/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png)](/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png)\n\n*Figure 4 from the paper, excerpted from [**page 2 of the v3 PDF**](https://arxiv.org/pdf/1610.02357v3#page=2). An “extreme” version of the Inception module, with one spatial convolution per output channel of the $1\\times1$ convolution.*\n\nLet's fix the channel count at 96 and see how the weights of the $3\\times3$ layer change as the number of segments goes from 1 to 96:\n\n```python\nimport torch.nn as nn\n\nC = 96\nfor groups in (1, 3, 4, 32, 96):\n    conv = nn.Conv2d(C, C, 3, padding=1, groups=groups, bias=False)\n    print(f\"groups={groups:>2}: channels per segment {C // groups:>2}, 3×3 conv parameters {conv.weight.numel():>6,}, weight shape {tuple(conv.weight.shape)}\")\n```\n\n```text\ngroups= 1: channels per segment 96, 3×3 conv parameters 82,944, weight shape (96, 96, 3, 3)\ngroups= 3: channels per segment 32, 3×3 conv parameters 27,648, weight shape (96, 32, 3, 3)\ngroups= 4: channels per segment 24, 3×3 conv parameters 20,736, weight shape (96, 24, 3, 3)\ngroups=32: channels per segment  3, 3×3 conv parameters  2,592, weight shape (96, 3, 3, 3)\ngroups=96: channels per segment  1, 3×3 conv parameters    864, weight shape (96, 1, 3, 3)\n```\n\nThe second number in the weight shape is how many input channels each kernel can see. `groups=1` is an ordinary convolution, with every kernel seeing all 96 channels; at `groups=96`, each kernel sees only one channel, leaving just the spatial $3\\times3$.\n\nThe author views this whole series as a spectrum. At one end is an ordinary convolution (preceded by a $1\\times1$), with a single segment; at the other end, there is one segment per channel. Inception modules sit in between, splitting a few hundred channels into 3 or 4 segments.\n\n### The Other End of the Spectrum\n\nThe author points out that the extreme module at the other end of the spectrum is almost identical to an operation that already existed, the **depthwise separable convolution**. It works in two steps:\n\n1. **Depthwise convolution**: each input channel gets its own spatial convolution, with no exchange between channels.\n2. **Pointwise convolution**: a $1\\times1$ convolution projects the previous step's output onto a new channel space.\n\nThis is what TensorFlow and Keras call a separable convolution. Image processing also uses the name separable convolution, for spatial factorizations such as splitting a $7\\times7$ into a $1\\times7$ and a $7\\times1$, which Inception-v3 used; don't mix the two up.\n\n\n> [!NOTE]\n> \n> The depthwise separable convolution wasn't invented in this paper, and the author credits its origins. Laurent Sifre developed it during an internship at Google Brain in 2013 and used it in AlexNet, getting a small gain in accuracy, much faster convergence, and a much smaller model; the first layers of GoogLeNet and BN-Inception also used it. Around the same time, Andrew Howard at Google was using it to build MobileNets for mobile devices; see our [MobileNet-V1](/blog/papers/lightweight/mobilenet-v1) note.\n\n\nThere are still two small differences between the “extreme” Inception module and the depthwise separable convolution:\n\n1. **Order.** A depthwise separable convolution does the depthwise step first, then the $1\\times1$; Inception does the $1\\times1$ first. The author considers this unimportant, because networks are built by stacking layers, so one module's $1\\times1$ is followed by the next module's depthwise step, and the order differs only at the two ends.\n2. **Whether there's a nonlinearity in between.** In Inception, both steps are followed by ReLU; a depthwise separable convolution usually has no activation function between its two steps. The author thinks this one might matter and leaves it for the experiments.\n\n### Xception\n\nSince the Inception hypothesis says that “cross-channel and spatial relationships can be partly separated,” the author goes all the way with a stronger hypothesis, that the two can be **completely** separated. The network based on this hypothesis is built entirely from depthwise separable convolutions and is named Xception, after Extreme Inception.\n\n[![The Xception architecture in three columns. Entry flow: 299x299x3 images pass through Conv 32 3x3 stride 2 and Conv 64 3x3, then three modules, each with two SeparableConv layers (128, 256, and 728 channels) plus MaxPooling 3x3 stride 2, with a Conv 1x1 stride 2 shortcut alongside, outputting 19x19x728. Middle flow: three layers of ReLU plus SeparableConv 728, wrapped by an identity shortcut, repeated 8 times. Exit flow: SeparableConv 728, SeparableConv 1024, and MaxPooling, with a Conv 1x1 stride 2 shortcut alongside; then SeparableConv 1536, SeparableConv 2048, and GlobalAveragePooling to get 2048-dimensional vectors, optional fully connected layers, and finally Logistic regression](/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png)](/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png)\n\n*Figure 5 from the paper, excerpted from [**page 5 of the v3 PDF**](https://arxiv.org/pdf/1610.02357v3#page=5). Every Conv and SeparableConv is followed by BN (omitted from the figure), and the depthwise part of SeparableConv doesn't expand the channels.*\n\nData moves through three sections from left to right. The entry flow starts from a $299\\times299$ image; the first two layers are ordinary convolutions, followed by three modules, each of which halves the feature map and widens the channels, ending at $19\\times19\\times728$. The middle flow repeats the same module 8 times without changing the size. The exit flow shrinks it once more and raises the channels to 2048, and global average pooling is followed by the classification layer.\n\nThe whole network has 36 convolutional layers organized into 14 modules, and every module except the first and the last is wrapped by a residual connection. Modules that don't shrink the feature map use an identity shortcut; those that do put a stride-2 $1\\times1$ convolution on the shortcut.\n\nCompared with Inception-v3, this diagram is much simpler, just depthwise separable convolutions and residual connections stacked layer by layer; the author says it takes only 30 to 40 lines in Keras. Let's build it in PyTorch following the figure:\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass SeparableConv(nn.Sequential):\n    \"\"\"Per-channel 3×3 convolution followed by a 1×1 convolution; no activation function in between, BN at the end\"\"\"\n    def __init__(self, cin, cout):\n        super().__init__(\n            nn.Conv2d(cin, cin, 3, padding=1, groups=cin, bias=False),   # depthwise\n            nn.Conv2d(cin, cout, 1, bias=False),                          # pointwise\n            nn.BatchNorm2d(cout))\n\nclass Block(nn.Module):\n    def __init__(self, cin, widths, relu_first=True, downsample=True):\n        super().__init__()\n        layers, c = [], cin\n        for i, w in enumerate(widths):\n            if relu_first or i > 0:\n                layers.append(nn.ReLU())\n            layers.append(SeparableConv(c, w))\n            c = w\n        if downsample:\n            layers.append(nn.MaxPool2d(3, 2, padding=1))\n            self.shortcut = nn.Sequential(nn.Conv2d(cin, c, 1, 2, bias=False), nn.BatchNorm2d(c))\n        else:\n            self.shortcut = nn.Identity()\n        self.body = nn.Sequential(*layers)\n\n    def forward(self, x):\n        return self.body(x) + self.shortcut(x)\n\ndef conv_bn_relu(cin, cout, stride):\n    return nn.Sequential(nn.Conv2d(cin, cout, 3, stride, bias=False), nn.BatchNorm2d(cout), nn.ReLU())\n\nentry = nn.Sequential(conv_bn_relu(3, 32, 2), conv_bn_relu(32, 64, 1),\n                      Block(64, [128, 128], relu_first=False), Block(128, [256, 256]), Block(256, [728, 728]))\nmiddle = nn.Sequential(*[Block(728, [728, 728, 728], downsample=False) for _ in range(8)])\nexit_ = nn.Sequential(Block(728, [728, 1024]),\n                      SeparableConv(1024, 1536), nn.ReLU(), SeparableConv(1536, 2048), nn.ReLU(),\n                      nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(2048, 1000))\nmodel = nn.Sequential(entry, middle, exit_)\n\nx = torch.randn(1, 3, 299, 299)\nfor name, part in [(\"entry flow\", entry), (\"middle flow\", middle)]:\n    x = part(x)\n    print(f\"After {name}:\", tuple(x.shape[1:]))\nprint(\"After exit flow:\", tuple(exit_(x).shape[1:]))\n\nparams = sum(p.numel() for p in model.parameters())\nprint(f\"Parameters: {params:,}\")\n```\n\n```text\nAfter entry flow: (728, 19, 19)\nAfter middle flow: (728, 19, 19)\nAfter exit flow: (1000,)\nParameters: 22,855,952\n```\n\nThe parameter count is exactly the 22,855,952 reported in the paper. PyTorch has no ready-made separable convolution layer; as above, a $3\\times3$ convolution with `groups` set to the number of input channels, followed by a $1\\times1$ convolution, does the job. Keras has `SeparableConv2D` built in, and Xception itself is included in Keras Applications.\n\nFollowing the details in the figure, each SeparableConv has a ReLU in front and BN behind, with nothing between the depthwise and pointwise steps. The first SeparableConv of the entry flow's first residual module has no ReLU in front, because the Conv 64 before it already ends with a ReLU; in the last two layers of the exit flow, the ReLU comes after.\n\n## Discussion\n\nThe author compares against Inception-v3 because the two have almost the same number of parameters, so any gap can't be attributed to model capacity. The comparison is run on two datasets:\n\n- **ImageNet**: single-label classification over 1000 classes.\n- **JFT**: an internal Google dataset of over 350 million high-resolution images in 17,000 classes, where each image can have multiple labels. Evaluation uses a separate set of 14,000 images, FastEval14k, averaging 36.5 labels per image; the metric is MAP@100 over the top 100 predictions, weighted by how common each class is on social media.\n\nBoth networks use the same training setup, one carefully tuned for Inception-v3, and the author didn't tune it separately for Xception. The one exception is weight decay, where Inception-v3's 4e-5 is clearly poor for Xception, so 1e-5 is used instead. All models are trained on 60 NVIDIA K80 GPUs; each ImageNet run takes about 3 days, and each JFT run over a month, without being trained to convergence.\n\n### ImageNet\n\n| Network | Top-1 accuracy | Top-5 accuracy |\n| --- | ---: | ---: |\n| VGG-16 | 0.715 | 0.901 |\n| ResNet-152 | 0.770 | 0.933 |\n| Inception-v3 | 0.782 | 0.941 |\n| **Xception** | **0.790** | **0.945** |\n\n*Table: Excerpted from Table 1 of the paper. Single-model, single-crop results on the validation set. Inception-v3 doesn't use the auxiliary classifier.*\n\nXception is 0.8 percentage points higher than Inception-v3, a small gap. VGG-16 and ResNet-152 are there only for reference.\n\n### JFT\n\n| Network | FastEval14k MAP@100 |\n| --- | ---: |\n| Inception-v3, no fully connected layers | 6.36 |\n| Xception, no fully connected layers | 6.70 |\n| Inception-v3, with fully connected layers | 6.50 |\n| **Xception, with fully connected layers** | **6.78** |\n\n*Table: Excerpted from Table 2 of the paper, results after 30 million iterations (about a month) of training. “With fully connected layers” means two 4096-dimensional fully connected layers are added before the classification layer.*\n\nOn JFT the gap is much larger, a 4.3% relative improvement for the versions with fully connected layers. The author's explanation is that Inception-v3 was developed with ImageNet in mind, so its design may already be overfit to that dataset, while neither network was tuned for JFT.\n\n### Similar Parameters, a Bit Slower\n\n| Network | Parameters | Training steps per second |\n| --- | ---: | ---: |\n| Inception-v3 | 23,626,728 | 31 |\n| Xception | 22,855,952 | 28 |\n\n*Table: Excerpted from Table 3 of the paper, with synchronous training on ImageNet on 60 K80 GPUs.*\n\nThe two differ in parameter count by less than 3.5%, so the author says Xception's gains come from using its parameters more efficiently, not from added capacity.\n\nIn speed, Xception is actually a little slower. Depthwise convolutions have few parameters and little computation, but on GPUs it's hard to get a matching speedup out of them.\n\n### Residual Connections\n\nThe author also trained a version of Xception with all the residual connections removed. On ImageNet, the version without residuals converges much more slowly and ends at clearly lower accuracy.\n\nThe author also cautions that the non-residual version used the same training setup, which may not be quite fair to it, and that this only shows residual connections matter for **this particular architecture**. Separately, he tried VGG-style networks with every convolution replaced by a depthwise separable convolution and no residuals at all, and on JFT they likewise beat an Inception-v3 with the same parameter count.\n\n### An Activation in Between?\n\nBack to the question left open earlier. Should there be a nonlinearity between the depthwise and pointwise steps? The author tried three options, none, ReLU, and ELU:\n\n[![Training curves of ImageNet validation accuracy against gradient descent steps. The red No intermediate activation rises fastest and ends near 0.79; the green Intermediate ELU comes next, leveling off around 0.78; the blue Intermediate ReLU is slowest, leveling off around 0.77](/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png)](/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png)\n\n*Figure 10 from the paper, excerpted from [**page 7 of the v3 PDF**](https://arxiv.org/pdf/1610.02357v3#page=7). Training curves with different activation functions between the depthwise and pointwise steps of SeparableConv.*\n\nThe red line, with no activation function at all, converges fastest and ends with the highest accuracy; the green line with ELU is in the middle, and the blue line with ReLU is the worst.\n\nThis is the opposite of what the Inception-v3 paper found. When Inception-v3 factorized a $5\\times5$ into two $3\\times3$ layers, it found that adding ReLU in between worked better. The author's guess is that the key lies in how “thick” the feature space is in which the spatial convolution operates. In an Inception module, each segment has dozens to hundreds of channels, and the nonlinearity helps; in a depthwise separable convolution, each segment has just one channel, and when ReLU chops off the negative values, information may simply be lost.\n\nMobileNetV2 later removed ReLU from the narrowest layers of the network, on the same grounds that “nonlinearities in low-dimensional spaces lose information”; you can continue with [MobileNet-V2](/blog/papers/lightweight/mobilenet-v2).\n\n## Conclusion\n\nXception looks at Inception from a new angle. Simplified, an Inception module is just a $1\\times1$ convolution plus a $3\\times3$ convolution split into a few groups; turn the group count from 1 up to the number of channels, and you go from an ordinary convolution to a depthwise separable convolution. Xception goes straight to the end, with almost the same parameter count as Inception-v3, slightly better on ImageNet and quite a bit better on JFT.\n\nIt may be called Extreme Inception, but there's nothing extreme about its speed; it even manages 3 fewer training steps per second than Inception-v3. The author expected it to pull ahead once depthwise convolution implementations were optimized. That wait lasted years, and [EfficientNetV2](/blog/papers/classic-cnns/efficientnet-v2) eventually just replaced the depthwise convolutions in the early part of the network.\n\nAs for whether the end is the best place to be, even the author wasn't sure; in his closing section he writes that some points in the middle of the spectrum might be better. A little over a month later, [ResNeXt](/blog/papers/classic-cnns/resnext) used convolutions split into 32 groups, landing right in the middle.\n","imageSizes":{"/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png":{"width":917,"height":554},"/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png":{"width":917,"height":538},"/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png":{"width":2054,"height":1417},"/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png":{"width":946,"height":716}}}