{"notes":[{"id":"classic-cnns/xception","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Xception","subtitle":"Inception 的尽头","author":"Z. Yuan","paperDate":"2016-10-07","publishedOn":"2024-03-20","world":"external/xception","paperTitle":"Xception: Deep Learning with Depthwise Separable Convolutions","paperUrl":"https://arxiv.org/abs/1610.02357","bibliography":{"title":"Xception: Deep Learning with Depthwise Separable Convolutions","authors":["François Chollet"],"url":"https://arxiv.org/abs/1610.02357","pdfUrl":"https://arxiv.org/pdf/1610.02357v3","codeUrl":null,"preprint":{"id":"1610.02357","submittedOn":"2016-10-07","url":"https://arxiv.org/abs/1610.02357"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2017/html/Chollet_Xception_Deep_Learning_CVPR_2017_paper.html"},"citation":"@misc{classiccnnsxception2016,\n  title = {{Xception: Deep Learning with Depthwise Separable Convolutions}},\n  author = {François Chollet},\n  year = {2016},\n  eprint = {1610.02357},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1610.02357}\n}\n","citationPath":"/content/papers/citations/classic-cnns/xception.3a6fa7188c36.bib"},"thumbnail":{"src":"/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png","width":917,"height":538},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/xception.46a958dbdfb9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"换个角度看-inception-模块","text":"换个角度看 Inception 模块","depth":3},{"id":"一路分到底","text":"一路分到底","depth":3},{"id":"光谱的另一端","text":"光谱的另一端","depth":3},{"id":"xception","text":"Xception","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"jft","text":"JFT","depth":3},{"id":"参数差不多速度慢一点","text":"参数差不多，速度慢一点","depth":3},{"id":"残差连接","text":"残差连接","depth":3},{"id":"中间要不要加激活函数","text":"中间要不要加激活函数","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"lightweight/mobilenet-v1","relation":"referenced","topic":""}]},{"id":"classic-cnns/inception-v1","category":"classic-cnns","legacyPath":"classic-cnns/1409-inception-v1","legacySlug":"1409-inception-v1","legacyPaths":[],"title":"GoogLeNet","subtitle":"或叫做 Inception-V1","author":"Z. Yuan","paperDate":"2014-09-17","publishedOn":"2024-09-22","paperTitle":"Going Deeper with Convolutions","paperUrl":"https://arxiv.org/abs/1409.4842","bibliography":{"title":"Going Deeper with Convolutions","authors":["Christian Szegedy","Wei Liu","Yangqing Jia","Pierre Sermanet","Scott Reed","Dragomir Anguelov","Dumitru Erhan","Vincent Vanhoucke","Andrew Rabinovich"],"url":"https://arxiv.org/abs/1409.4842","pdfUrl":"https://arxiv.org/pdf/1409.4842","codeUrl":null,"preprint":{"id":"1409.4842","submittedOn":"2014-09-17","url":"https://arxiv.org/abs/1409.4842"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2015.7298594"},"citation":"@misc{classiccnnsinceptionv12014,\n  title = {{Going Deeper with Convolutions}},\n  author = {Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich},\n  year = {2014},\n  eprint = {1409.4842},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1409.4842}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v1.3d20e0e32bcd.bib"},"thumbnail":{"src":"/content/papers/assets/254b970b1374c24267b60c2d684338760e9ba8b10dad761a5fa15f1bb8bcf6ef.jpg","width":1330,"height":394},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inception-v1.7c8c2ff19589.md","headings":[{"id":"解决问题","text":"解决问题","depth":2},{"id":"inception-模块","text":"Inception 模块","depth":3},{"id":"参数配置","text":"参数配置","depth":3},{"id":"模型架构","text":"模型架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"在-ilsvrc-2014-上的表现","text":"在 ILSVRC 2014 上的表现","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分组卷积与特征复用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分组卷积与特征复用"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"大卷积核退场","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2023-11-15","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html"},"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inception-v3.2087fa7b4a8e.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"把-55-拆成两层-33","text":"把 5×5 拆成两层 3×3","depth":3},{"id":"再拆成-1n-与-n1","text":"再拆成 1×n 与 n×1","depth":3},{"id":"辅助分类器的真正作用","text":"辅助分类器的真正作用","depth":3},{"id":"缩小特征图又不想卡住信息","text":"缩小特征图，又不想卡住信息","depth":3},{"id":"把零件组装起来","text":"把零件组装起来","depth":3},{"id":"让模型别太有把握","text":"让模型别太有把握","depth":3},{"id":"训练设置","text":"训练设置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"一项一项加上去","text":"一项一项加上去","depth":3},{"id":"分辨率低一点差多少","text":"分辨率低一点，差多少？","depth":3},{"id":"和其他模型比较","text":"和其他模型比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]},{"id":"lightweight/mobilenet-v1","category":"lightweight","legacyPath":"lightweight/1704-mobilenet-v1","legacySlug":"1704-mobilenet-v1","legacyPaths":[],"title":"MobileNet-V1","subtitle":"深度可分离卷积的发扬者","author":"Z. Yuan","paperDate":"2017-04-17","publishedOn":"2024-06-25","paperTitle":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","paperUrl":"https://arxiv.org/abs/1704.04861","bibliography":{"title":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","authors":["Andrew G. Howard","Menglong Zhu","Bo Chen","Dmitry Kalenichenko","Weijun Wang","Tobias Weyand","Marco Andreetto","Hartwig Adam"],"url":"https://arxiv.org/abs/1704.04861","pdfUrl":"https://arxiv.org/pdf/1704.04861","codeUrl":null,"preprint":{"id":"1704.04861","submittedOn":"2017-04-17","url":"https://arxiv.org/abs/1704.04861"},"publication":null,"citation":"@misc{lightweightmobilenetv12017,\n  title = {{MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications}},\n  author = {Andrew G. Howard and Menglong Zhu and Bo Chen and Dmitry Kalenichenko and Weijun Wang and Tobias Weyand and Marco Andreetto and Hartwig Adam},\n  year = {2017},\n  eprint = {1704.04861},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1704.04861}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v1.e9c4c525fed4.bib"},"thumbnail":{"src":"/content/papers/assets/b8390bd8e0f21f1b1eaa6b2a5d4a1b0d9eba36e2fe5dd0878830d757839d30f1.jpg","width":1064,"height":1080},"locale":"zh-hans","path":"/content/papers/zh-hans/lightweight/mobilenet-v1.aef841466ce6.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"深度可分离卷积","text":"深度可分离卷积","depth":3},{"id":"网络架构","text":"网络架构","depth":2},{"id":"训练技巧","text":"训练技巧","depth":2},{"id":"讨论","text":"讨论","depth":2},{"id":"深度可分离卷积的性能","text":"深度可分离卷积的性能","depth":3},{"id":"要深度还是要宽度","text":"要深度还是要宽度？","depth":3},{"id":"和其他模型比较","text":"和其他模型比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v2","relation":"next","topic":"移动设备上的高效卷积"}]}],"text":"到了 2016 年，Inception 已经出到第四代，从 GoogLeNet、加了 BN 的 Inception-v2、分解卷积的 Inception-v3，一路到接上残差的 Inception-ResNet。每一代都在 ImageNet 上名列前茅，模块却一代比一代复杂，分支、通道数、卷积核形状都要逐一调整。\n\n\n> [!TIP]\n> \n> Inception 系列可以先看这两篇：\n> \n> - **[[14.09] GoogLeNet：或叫做 Inception-V1](/blog/papers/classic-cnns/inception-v1)**\n> - **[[15.12] Inception-v3：大卷积核退场](/blog/papers/classic-cnns/inception-v3)**\n\n\nFrançois Chollet 是 Keras 的作者，当时在 Google。他想弄清楚 Inception 模块为什么有效、和普通的卷积差在哪里，以及 Inception 之后还能往哪里走。\n\n这篇论文就是他的回答。\n\n## 定义问题\n\n普通的卷积层，每个卷积核都是一块立体的滤波器，宽、高之外还有通道这个维度。一个 $3\\times3$ 的卷积核看 256 个输入通道，就有 $3\\times3\\times256$ 个权重，它**同时**处理同一个位置上不同通道之间的关系（跨通道相关性），以及相邻位置之间的关系（空间相关性）。\n\nInception 模块把这两件事拆开做。它先用几个 $1\\times1$ 卷积处理跨通道的关系，把输入投影到 3 到 4 个比较小的空间，再在各个小空间里用 $3\\times3$ 或 $5\\times5$ 卷积处理空间关系。\n\n作者把这个设计背后的想法叫做 **Inception 假设**，也就是跨通道相关性和空间相关性足够独立，分开处理比一起处理好。\n\n如果「拆开一部分」有效，那拆得更彻底呢？\n\n## 解决问题\n\n### 换个角度看 Inception 模块\n\n作者先把 Inception 模块简化，只保留一种 $3\\times3$ 卷积，去掉平均池化那条分支。剩下三条分支，每条都是先 $1\\times1$、再 $3\\times3$。\n\n这三个 $1\\times1$ 卷积看的是同一个输入，所以可以合并成一个输出通道比较多的 $1\\times1$ 卷积。合并之后，三个 $3\\times3$ 卷积就变成各自处理这些输出通道里的其中一段：\n\n[![简化 Inception 模块的等价形式，从下往上：Input 进入一个 1x1 conv，输出通道（Output channels）被大括号分成三段，每段各接一个 3x3 conv，最后 Concat](/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png)](/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png)\n\n*原论文 Figure 3，取自 [**v3 PDF 第 2 页**](https://arxiv.org/pdf/1610.02357v3#page=2)。和简化 Inception 模块严格等价的写法。*\n\n这两种写法真的一样吗？我们拿一个 64 通道的输入试试。三条分支各自做 $1\\times1$（64 到 32）和 $3\\times3$；另一边用一个 64 到 96 的 $1\\times1$ 卷积，接着一个「把 96 个通道分成三组，各组自己做 $3\\times3$」的卷积，权重从三条分支直接搬过来：\n\n```python\nimport torch\nimport torch.nn as nn\n\ntorch.manual_seed(0)\nx = torch.randn(1, 64, 8, 8)                       # 64 个通道的输入\n\n# Figure 2：三条分支，各自先 1×1 再 3×3\npw = [nn.Conv2d(64, 32, 1, bias=False) for _ in range(3)]\nsp = [nn.Conv2d(32, 32, 3, padding=1, bias=False) for _ in range(3)]\ny2 = torch.cat([s(p(x)) for p, s in zip(pw, sp)], dim=1)\n\n# Figure 3：一个大的 1×1 卷积，接着把 96 个输出通道切成三段，各自做 3×3\nbig_pw = nn.Conv2d(64, 96, 1, bias=False)\ngrouped = nn.Conv2d(96, 96, 3, padding=1, groups=3, bias=False)\nwith torch.no_grad():\n    big_pw.weight.copy_(torch.cat([p.weight for p in pw]))\n    grouped.weight.copy_(torch.cat([s.weight for s in sp]))\ny3 = grouped(big_pw(x))\n\nprint(\"输出形状：\", tuple(y2.shape), tuple(y3.shape))\nprint(\"两种写法的最大差异：\", (y2 - y3).abs().max().item())\n```\n\n```text\n输出形状： (1, 96, 8, 8) (1, 96, 8, 8)\n两种写法的最大差异： 7.748603820800781e-07\n```\n\n差异只有浮点数的舍入误差。PyTorch 的 `groups=3` 就是「把通道分成三段，各段各自卷积」，所以这个简化的 Inception 模块，其实就是一个 $1\\times1$ 卷积加上一个分成三组的 $3\\times3$ 卷积。\n\n这样看，简化后的 Inception 模块只剩一个可以调的数字，也就是通道分成几段。\n\n### 一路分到底\n\n段数越多，每段的通道越少。分到极端，每一段只剩一个通道，每个输出通道都配一个自己的 $3\\times3$ 卷积：\n\n[![极端版本的 Inception 模块，从下往上：Input 进入一个 1x1 conv，输出通道被分成很多段，每段只接一个 3x3，最后 Concat](/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png)](/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png)\n\n*原论文 Figure 4，取自 [**v3 PDF 第 2 页**](https://arxiv.org/pdf/1610.02357v3#page=2)。「极端」版本的 Inception 模块，$1\\times1$ 卷积的每个输出通道各有一个空间卷积。*\n\n我们固定 96 个通道，看段数从 1 加到 96 时，$3\\times3$ 那一层的权重怎么变：\n\n```python\nimport torch.nn as nn\n\nC = 96\nfor groups in (1, 3, 4, 32, 96):\n    conv = nn.Conv2d(C, C, 3, padding=1, groups=groups, bias=False)\n    print(f\"groups={groups:>2}：每段 {C // groups:>2} 个通道，3×3 卷积的参数 {conv.weight.numel():>6,}，权重形状 {tuple(conv.weight.shape)}\")\n```\n\n```text\ngroups= 1：每段 96 个通道，3×3 卷积的参数 82,944，权重形状 (96, 96, 3, 3)\ngroups= 3：每段 32 个通道，3×3 卷积的参数 27,648，权重形状 (96, 32, 3, 3)\ngroups= 4：每段 24 个通道，3×3 卷积的参数 20,736，权重形状 (96, 24, 3, 3)\ngroups=32：每段  3 个通道，3×3 卷积的参数  2,592，权重形状 (96, 3, 3, 3)\ngroups=96：每段  1 个通道，3×3 卷积的参数    864，权重形状 (96, 1, 3, 3)\n```\n\n权重形状的第二个数字，就是每个卷积核看得到几个输入通道。`groups=1` 是普通的卷积，每个卷积核看全部 96 个通道；`groups=96` 时，每个卷积核只看一个通道，只剩下空间方向的 $3\\times3$。\n\n作者把这一整排看成一条光谱。一端是普通的卷积（前面接一个 $1\\times1$），只有一段；另一端是每个通道一段。Inception 模块在中间，把几百个通道分成 3 或 4 段。\n\n### 光谱的另一端\n\n作者指出，光谱另一端的极端模块，几乎就是已经存在的**深度可分离卷积**（depthwise separable convolution）。它分成两步：\n\n1. **depthwise 卷积**：每个输入通道各自做空间卷积，通道之间不交流。\n2. **pointwise 卷积**：用 $1\\times1$ 卷积把上一步的输出投影到新的通道空间。\n\nTensorFlow、Keras 里叫做 separable convolution 的就是它。图像处理领域也有 separable convolution 这个名字，指的是把 $7\\times7$ 拆成 $1\\times7$ 和 $7\\times1$ 这种空间上的拆法，Inception-v3 就用过，两者不要搞混。\n\n\n> [!NOTE]\n> \n> 深度可分离卷积不是这篇发明的。作者在论文里交代了来源，Laurent Sifre 2013 年在 Google Brain 实习时开发了它，用在 AlexNet 上，准确率小幅提升，收敛快了很多，模型也小了很多；GoogLeNet 和 BN-Inception 的第一层也用过。与此同时，Google 的 Andrew Howard 正在用它做移动设备上的 MobileNets，可以看我们的 [MobileNet-V1](/blog/papers/lightweight/mobilenet-v1) 笔记。\n\n\n「极端」的 Inception 模块和深度可分离卷积，还有两个小差别：\n\n1. **顺序**。深度可分离卷积先做 depthwise、再做 $1\\times1$；Inception 先做 $1\\times1$。作者认为这不重要，网络是一层层叠起来的，前一个模块的 $1\\times1$ 接着下一个模块的 depthwise，顺序只差在头尾。\n2. **中间有没有非线性**。Inception 的两步后面都接 ReLU；深度可分离卷积通常两步之间不加激活函数。作者认为这一点可能有影响，留到实验验证。\n\n### Xception\n\n既然 Inception 假设是「跨通道和空间关系可以部分分开」，作者干脆做了一个更强的假设，认为两者可以**完全**分开。基于这个假设的网络，全部用深度可分离卷积搭成，名字取自 Extreme Inception，叫做 Xception。\n\n[![Xception 的架构图，分成三栏。Entry flow：299x299x3 图像，经过 Conv 32 3x3 stride 2、Conv 64 3x3，接着三个模块，每个是两层 SeparableConv（128、256、728 个通道）加 MaxPooling 3x3 stride 2，旁边是 Conv 1x1 stride 2 的捷径，输出 19x19x728。Middle flow：三层 ReLU 加 SeparableConv 728，外面一条恒等捷径，重复 8 次。Exit flow：SeparableConv 728、SeparableConv 1024、MaxPooling，旁边是 Conv 1x1 stride 2 捷径；接着 SeparableConv 1536、SeparableConv 2048、GlobalAveragePooling 得到 2048 维向量，可选的全连接层，最后是 Logistic regression](/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png)](/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png)\n\n*原论文 Figure 5，取自 [**v3 PDF 第 5 页**](https://arxiv.org/pdf/1610.02357v3#page=5)。所有 Conv 和 SeparableConv 后面都接 BN（图上省略），SeparableConv 的 depthwise 部分不扩增通道。*\n\n数据从左往右走三段。entry flow 从 $299\\times299$ 的图像开始，前两层是普通卷积，接着三个模块，每个模块把特征图缩小一半、通道加宽，最后停在 $19\\times19\\times728$。middle flow 是同一个模块重复 8 次，大小不变。exit flow 再缩小一次，把通道加到 2048，全局平均池化后接分类层。\n\n整个网络有 36 层卷积，分成 14 个模块，除了第一个和最后一个模块，每个模块外面都有一条残差连接。不需要缩小的模块用恒等捷径；需要缩小的，捷径上放一个步幅 2 的 $1\\times1$ 卷积。\n\n和 Inception-v3 比，这张图简单很多，就是深度可分离卷积和残差连接一层层叠起来，作者说用 Keras 写只要 30 到 40 行。我们照图用 PyTorch 搭一遍：\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass SeparableConv(nn.Sequential):\n    \"\"\"逐通道 3×3 卷积，接着 1×1 卷积；两者之间没有激活函数，最后接 BN\"\"\"\n    def __init__(self, cin, cout):\n        super().__init__(\n            nn.Conv2d(cin, cin, 3, padding=1, groups=cin, bias=False),   # depthwise\n            nn.Conv2d(cin, cout, 1, bias=False),                          # pointwise\n            nn.BatchNorm2d(cout))\n\nclass Block(nn.Module):\n    def __init__(self, cin, widths, relu_first=True, downsample=True):\n        super().__init__()\n        layers, c = [], cin\n        for i, w in enumerate(widths):\n            if relu_first or i > 0:\n                layers.append(nn.ReLU())\n            layers.append(SeparableConv(c, w))\n            c = w\n        if downsample:\n            layers.append(nn.MaxPool2d(3, 2, padding=1))\n            self.shortcut = nn.Sequential(nn.Conv2d(cin, c, 1, 2, bias=False), nn.BatchNorm2d(c))\n        else:\n            self.shortcut = nn.Identity()\n        self.body = nn.Sequential(*layers)\n\n    def forward(self, x):\n        return self.body(x) + self.shortcut(x)\n\ndef conv_bn_relu(cin, cout, stride):\n    return nn.Sequential(nn.Conv2d(cin, cout, 3, stride, bias=False), nn.BatchNorm2d(cout), nn.ReLU())\n\nentry = nn.Sequential(conv_bn_relu(3, 32, 2), conv_bn_relu(32, 64, 1),\n                      Block(64, [128, 128], relu_first=False), Block(128, [256, 256]), Block(256, [728, 728]))\nmiddle = nn.Sequential(*[Block(728, [728, 728, 728], downsample=False) for _ in range(8)])\nexit_ = nn.Sequential(Block(728, [728, 1024]),\n                      SeparableConv(1024, 1536), nn.ReLU(), SeparableConv(1536, 2048), nn.ReLU(),\n                      nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(2048, 1000))\nmodel = nn.Sequential(entry, middle, exit_)\n\nx = torch.randn(1, 3, 299, 299)\nfor name, part in [(\"entry flow\", entry), (\"middle flow\", middle)]:\n    x = part(x)\n    print(f\"{name} 之后：\", tuple(x.shape[1:]))\nprint(\"exit flow 之后：\", tuple(exit_(x).shape[1:]))\n\nparams = sum(p.numel() for p in model.parameters())\nprint(f\"参数量：{params:,}\")\n```\n\n```text\nentry flow 之后： (728, 19, 19)\nmiddle flow 之后： (728, 19, 19)\nexit flow 之后： (1000,)\n参数量：22,855,952\n```\n\n参数量和论文报告的 22,855,952 完全一样。PyTorch 没有现成的 separable convolution 层，照这样写一个 `groups=输入通道数` 的 $3\\times3$ 卷积接一个 $1\\times1$ 卷积就行；Keras 则直接有 `SeparableConv2D`，Xception 本身也收录在 Keras Applications 里。\n\n对照图上的细节，每个 SeparableConv 前面放 ReLU、后面接 BN，depthwise 和 pointwise 之间什么都不加。entry flow 第一个残差模块的第一层 SeparableConv 前面没有 ReLU，因为前面的 Conv 64 已经接了 ReLU；exit flow 最后两层则是 ReLU 放在后面。\n\n## 讨论\n\n作者拿 Inception-v3 来比，因为两者的参数量几乎一样，差距不能归因于模型容量。比较在两个数据集上进行：\n\n- **ImageNet**：1000 类的单标签分类。\n- **JFT**：Google 内部的数据集，超过 3.5 亿张高分辨率图像，17,000 个类别，每张可以有多个标签。评估用另一个 14,000 张的 FastEval14k，平均每张 36.5 个标签，指标是前 100 个预测的 MAP@100，并按各类别在社交媒体上的常见程度加权。\n\n两个网络用的是同一套训练设置，这套设置是为 Inception-v3 仔细调过的，作者没有为 Xception 另外调。唯一的例外是 weight decay，Inception-v3 的 4e-5 对 Xception 明显不好，改用 1e-5。所有模型都在 60 张 NVIDIA K80 上训练，ImageNet 每次大约 3 天，JFT 每次超过一个月，而且没有训练到收敛。\n\n### ImageNet\n\n| 网络 | Top-1 准确率 | Top-5 准确率 |\n| --- | ---: | ---: |\n| VGG-16 | 0.715 | 0.901 |\n| ResNet-152 | 0.770 | 0.933 |\n| Inception-v3 | 0.782 | 0.941 |\n| **Xception** | **0.790** | **0.945** |\n\n*表：摘自原论文 Table 1，单模型、单次裁剪的验证集结果。Inception-v3 没有使用辅助分类器。*\n\nXception 比 Inception-v3 高 0.8 个百分点，差距不大。VGG-16 和 ResNet-152 只是放着参考。\n\n### JFT\n\n| 网络 | FastEval14k MAP@100 |\n| --- | ---: |\n| Inception-v3，无全连接层 | 6.36 |\n| Xception，无全连接层 | 6.70 |\n| Inception-v3，有全连接层 | 6.50 |\n| **Xception，有全连接层** | **6.78** |\n\n*表：摘自原论文 Table 2，训练 3,000 万次迭代（约一个月）后的结果。「有全连接层」指在分类层前加两层 4096 维的全连接层。*\n\n在 JFT 上差距大得多，有全连接层的版本相对提升了 4.3%。作者的解释是，Inception-v3 是针对 ImageNet 发展出来的，设计上可能已经过度贴合这个数据集；JFT 则是两个网络都没调过。\n\n### 参数差不多，速度慢一点\n\n| 网络 | 参数量 | 每秒训练步数 |\n| --- | ---: | ---: |\n| Inception-v3 | 23,626,728 | 31 |\n| Xception | 22,855,952 | 28 |\n\n*表：摘自原论文 Table 3，在 ImageNet 上以 60 张 K80 同步训练。*\n\n两者的参数量相差不到 3.5%，作者因此说，Xception 的进步来自更有效率地使用参数，而不是容量变大。\n\n速度上，Xception 反而慢一点。depthwise 卷积的参数和计算量虽然少，在 GPU 上却不容易跑出对应的速度。\n\n### 残差连接\n\n作者也训练了一个去掉所有残差连接的 Xception。在 ImageNet 上，没有残差的版本收敛慢很多，最后的准确率也明显比较低。\n\n作者也提醒，没有残差的版本用的是同一套训练设置，对它可能不太公平；而且这只说明残差连接对**这个架构**很重要。他另外试过把 VGG 式网络的卷积全部换成深度可分离卷积、完全不用残差，在 JFT 上一样赢过参数量相同的 Inception-v3。\n\n### 中间要不要加激活函数\n\n回到前面留下的问题，depthwise 和 pointwise 之间要不要加非线性？作者试了不加、加 ReLU、加 ELU 三种：\n\n[![ImageNet 验证准确率随梯度下降步数变化的训练曲线。红色的 No intermediate activation 上升最快，最后接近 0.79；绿色的 Intermediate ELU 次之，大约停在 0.78；蓝色的 Intermediate ReLU 最慢，大约停在 0.77](/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png)](/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png)\n\n*原论文 Figure 10，取自 [**v3 PDF 第 7 页**](https://arxiv.org/pdf/1610.02357v3#page=7)。在 SeparableConv 的 depthwise 与 pointwise 之间使用不同激活函数的训练曲线。*\n\n红线是不加任何激活函数，收敛最快，最后的准确率也最高；加 ELU 的绿线居中；加 ReLU 的蓝线最差。\n\n这和 Inception-v3 论文的结果相反。Inception-v3 把 $5\\times5$ 分解成两层 $3\\times3$ 时，发现中间加 ReLU 比较好。作者的推测是，关键在于空间卷积所在的特征空间有多「厚」。Inception 模块里，每段有几十上百个通道，非线性有帮助；深度可分离卷积的每段只有一个通道，ReLU 把负值砍掉，信息可能就这样丢了。\n\n后来的 MobileNetV2 在网络中最窄的那几层去掉了 ReLU，用的也是「在低维空间做非线性会丢信息」的理由，可以接着看 [MobileNet-V2](/blog/papers/lightweight/mobilenet-v2)。\n\n## 结论\n\nXception 换了一个角度看 Inception。简化之后的 Inception 模块，只是一个 $1\\times1$ 卷积加上分成几组的 $3\\times3$ 卷积；组数从 1 调到通道数，就从普通卷积走到了深度可分离卷积。Xception 直接走到尽头，参数量和 Inception-v3 几乎相同，ImageNet 上略好，JFT 上好得比较多。\n\n名字叫 Extreme Inception，速度倒是一点也不极端，每秒训练步数还比 Inception-v3 少 3 步。作者当时预期，depthwise 卷积的实现优化之后就会反超；这一等就是好几年，[EfficientNetV2](/blog/papers/classic-cnns/efficientnet-v2) 最后干脆把网络前段的 depthwise 卷积换掉了。\n\n至于尽头是不是最好的位置，作者自己也不确定，他在文末写道，光谱中间的某些点可能更好。一个多月后的 [ResNeXt](/blog/papers/classic-cnns/resnext) 用了分成 32 组的卷积，刚好落在中间。\n","imageSizes":{"/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png":{"width":917,"height":554},"/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png":{"width":917,"height":538},"/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png":{"width":2054,"height":1417},"/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png":{"width":946,"height":716}}}