{"notes":[{"id":"classic-cnns/xception","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Xception","subtitle":"Inception 的盡頭","author":"Z. Yuan","paperDate":"2016-10-07","publishedOn":"2024-03-20","world":"external/xception","paperTitle":"Xception: Deep Learning with Depthwise Separable Convolutions","paperUrl":"https://arxiv.org/abs/1610.02357","bibliography":{"title":"Xception: Deep Learning with Depthwise Separable Convolutions","authors":["François Chollet"],"url":"https://arxiv.org/abs/1610.02357","pdfUrl":"https://arxiv.org/pdf/1610.02357v3","codeUrl":null,"preprint":{"id":"1610.02357","submittedOn":"2016-10-07","url":"https://arxiv.org/abs/1610.02357"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2017/html/Chollet_Xception_Deep_Learning_CVPR_2017_paper.html"},"citation":"@misc{classiccnnsxception2016,\n  title = {{Xception: Deep Learning with Depthwise Separable Convolutions}},\n  author = {François Chollet},\n  year = {2016},\n  eprint = {1610.02357},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1610.02357}\n}\n","citationPath":"/content/papers/citations/classic-cnns/xception.3a6fa7188c36.bib"},"thumbnail":{"src":"/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png","width":917,"height":538},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/xception.68cdf9c18112.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"換個角度看-inception-模組","text":"換個角度看 Inception 模組","depth":3},{"id":"一路分到底","text":"一路分到底","depth":3},{"id":"光譜的另一端","text":"光譜的另一端","depth":3},{"id":"xception","text":"Xception","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"jft","text":"JFT","depth":3},{"id":"參數差不多速度慢一點","text":"參數差不多，速度慢一點","depth":3},{"id":"殘差連接","text":"殘差連接","depth":3},{"id":"中間要不要加激活函數","text":"中間要不要加激活函數","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"lightweight/mobilenet-v1","relation":"referenced","topic":""}]},{"id":"classic-cnns/inception-v1","category":"classic-cnns","legacyPath":"classic-cnns/1409-inception-v1","legacySlug":"1409-inception-v1","legacyPaths":[],"title":"GoogLeNet","subtitle":"或叫做 Inception-V1","author":"Z. Yuan","paperDate":"2014-09-17","publishedOn":"2024-09-22","paperTitle":"Going Deeper with Convolutions","paperUrl":"https://arxiv.org/abs/1409.4842","bibliography":{"title":"Going Deeper with Convolutions","authors":["Christian Szegedy","Wei Liu","Yangqing Jia","Pierre Sermanet","Scott Reed","Dragomir Anguelov","Dumitru Erhan","Vincent Vanhoucke","Andrew Rabinovich"],"url":"https://arxiv.org/abs/1409.4842","pdfUrl":"https://arxiv.org/pdf/1409.4842","codeUrl":null,"preprint":{"id":"1409.4842","submittedOn":"2014-09-17","url":"https://arxiv.org/abs/1409.4842"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2015.7298594"},"citation":"@misc{classiccnnsinceptionv12014,\n  title = {{Going Deeper with Convolutions}},\n  author = {Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich},\n  year = {2014},\n  eprint = {1409.4842},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1409.4842}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v1.3d20e0e32bcd.bib"},"thumbnail":{"src":"/content/papers/assets/254b970b1374c24267b60c2d684338760e9ba8b10dad761a5fa15f1bb8bcf6ef.jpg","width":1330,"height":394},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/inception-v1.91efdf81ddcf.md","headings":[{"id":"解決問題","text":"解決問題","depth":2},{"id":"inception-模塊","text":"Inception 模塊","depth":3},{"id":"參數配置","text":"參數配置","depth":3},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"在-ilsvrc-2014-上的表現","text":"在 ILSVRC 2014 上的表現","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分組卷積與特徵重用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分組卷積與特徵重用"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"大卷積核退場","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2023-11-15","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html"},"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/inception-v3.fb3811ac4302.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"把-55-拆成兩層-33","text":"把 5×5 拆成兩層 3×3","depth":3},{"id":"再拆成-1n-與-n1","text":"再拆成 1×n 與 n×1","depth":3},{"id":"輔助分類器的真正作用","text":"輔助分類器的真正作用","depth":3},{"id":"縮小特徵圖又不想卡住資訊","text":"縮小特徵圖，又不想卡住資訊","depth":3},{"id":"把零件組起來","text":"把零件組起來","depth":3},{"id":"讓模型別太有把握","text":"讓模型別太有把握","depth":3},{"id":"訓練設定","text":"訓練設定","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"一項一項加上去","text":"一項一項加上去","depth":3},{"id":"解析度低一點差多少","text":"解析度低一點，差多少？","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]},{"id":"lightweight/mobilenet-v1","category":"lightweight","legacyPath":"lightweight/1704-mobilenet-v1","legacySlug":"1704-mobilenet-v1","legacyPaths":[],"title":"MobileNet-V1","subtitle":"深度可分離卷積的發揚者","author":"Z. Yuan","paperDate":"2017-04-17","publishedOn":"2024-06-25","paperTitle":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","paperUrl":"https://arxiv.org/abs/1704.04861","bibliography":{"title":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","authors":["Andrew G. Howard","Menglong Zhu","Bo Chen","Dmitry Kalenichenko","Weijun Wang","Tobias Weyand","Marco Andreetto","Hartwig Adam"],"url":"https://arxiv.org/abs/1704.04861","pdfUrl":"https://arxiv.org/pdf/1704.04861","codeUrl":null,"preprint":{"id":"1704.04861","submittedOn":"2017-04-17","url":"https://arxiv.org/abs/1704.04861"},"publication":null,"citation":"@misc{lightweightmobilenetv12017,\n  title = {{MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications}},\n  author = {Andrew G. Howard and Menglong Zhu and Bo Chen and Dmitry Kalenichenko and Weijun Wang and Tobias Weyand and Marco Andreetto and Hartwig Adam},\n  year = {2017},\n  eprint = {1704.04861},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1704.04861}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v1.e9c4c525fed4.bib"},"thumbnail":{"src":"/content/papers/assets/b8390bd8e0f21f1b1eaa6b2a5d4a1b0d9eba36e2fe5dd0878830d757839d30f1.jpg","width":1064,"height":1080},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/mobilenet-v1.6d60a34b883b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"深度可分離卷積","text":"深度可分離卷積","depth":3},{"id":"網路架構","text":"網路架構","depth":2},{"id":"訓練技巧","text":"訓練技巧","depth":2},{"id":"討論","text":"討論","depth":2},{"id":"深度可分離卷積的效能","text":"深度可分離卷積的效能","depth":3},{"id":"要深度還是要寬度","text":"要深度還是要寬度？","depth":3},{"id":"和其他模型比較","text":"和其他模型比較","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v2","relation":"next","topic":"行動裝置上的高效率卷積"}]}],"text":"到了 2016 年，Inception 已經出到第四代，從 GoogLeNet、加了 BN 的 Inception-v2、分解卷積的 Inception-v3，一路到接上殘差的 Inception-ResNet。每一代都在 ImageNet 上名列前茅，模組卻一代比一代複雜，分支、通道數、卷積核形狀都要逐一調整。\n\n\n> [!TIP]\n> \n> Inception 系列可以先看這兩篇：\n> \n> - **[[14.09] GoogLeNet：或叫做 Inception-V1](/blog/papers/classic-cnns/inception-v1)**\n> - **[[15.12] Inception-v3：大卷積核退場](/blog/papers/classic-cnns/inception-v3)**\n\n\nFrançois Chollet 是 Keras 的作者，當時在 Google。他想弄清楚 Inception 模組為什麼有效、和一般的卷積差在哪裡，以及 Inception 之後還能往哪裡走。\n\n這篇論文就是他的回答。\n\n## 定義問題\n\n一般的卷積層，每個卷積核都是一塊立體的濾波器，寬、高之外還有通道這個維度。一個 $3\\times3$ 的卷積核看 256 個輸入通道，就有 $3\\times3\\times256$ 個權重，它**同時**處理同一個位置上不同通道之間的關係（跨通道相關性），以及相鄰位置之間的關係（空間相關性）。\n\nInception 模組把這兩件事拆開做。它先用幾個 $1\\times1$ 卷積處理跨通道的關係，把輸入投影到 3 到 4 個比較小的空間，再在各個小空間裡用 $3\\times3$ 或 $5\\times5$ 卷積處理空間關係。\n\n作者把這個設計背後的想法叫做 **Inception 假設**，也就是跨通道相關性和空間相關性夠獨立，分開處理比一起處理好。\n\n如果「拆開一部分」有效，那拆得更徹底呢？\n\n## 解決問題\n\n### 換個角度看 Inception 模組\n\n作者先把 Inception 模組簡化，只保留一種 $3\\times3$ 卷積，拿掉平均池化那條分支。剩下三條分支，每條都是先 $1\\times1$、再 $3\\times3$。\n\n這三個 $1\\times1$ 卷積看的是同一個輸入，所以可以合併成一個輸出通道比較多的 $1\\times1$ 卷積。合併之後，三個 $3\\times3$ 卷積就變成各自處理這些輸出通道裡的其中一段：\n\n[![簡化 Inception 模組的等價形式，由下往上：Input 進入一個 1x1 conv，輸出通道（Output channels）被大括號分成三段，每段各接一個 3x3 conv，最後 Concat](/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png)](/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png)\n\n*原論文 Figure 3，取自 [**v3 PDF 第 2 頁**](https://arxiv.org/pdf/1610.02357v3#page=2)。和簡化 Inception 模組嚴格等價的寫法。*\n\n這兩種寫法真的一樣嗎？我們拿一個 64 通道的輸入試試。三條分支各自做 $1\\times1$（64 到 32）和 $3\\times3$；另一邊用一個 64 到 96 的 $1\\times1$ 卷積，接著一個「把 96 個通道分成三組，各組自己做 $3\\times3$」的卷積，權重從三條分支直接搬過來：\n\n```python\nimport torch\nimport torch.nn as nn\n\ntorch.manual_seed(0)\nx = torch.randn(1, 64, 8, 8)                       # 64 個通道的輸入\n\n# Figure 2：三條分支，各自先 1×1 再 3×3\npw = [nn.Conv2d(64, 32, 1, bias=False) for _ in range(3)]\nsp = [nn.Conv2d(32, 32, 3, padding=1, bias=False) for _ in range(3)]\ny2 = torch.cat([s(p(x)) for p, s in zip(pw, sp)], dim=1)\n\n# Figure 3：一個大的 1×1 卷積，接著把 96 個輸出通道切成三段，各自做 3×3\nbig_pw = nn.Conv2d(64, 96, 1, bias=False)\ngrouped = nn.Conv2d(96, 96, 3, padding=1, groups=3, bias=False)\nwith torch.no_grad():\n    big_pw.weight.copy_(torch.cat([p.weight for p in pw]))\n    grouped.weight.copy_(torch.cat([s.weight for s in sp]))\ny3 = grouped(big_pw(x))\n\nprint(\"輸出形狀：\", tuple(y2.shape), tuple(y3.shape))\nprint(\"兩種寫法的最大差異：\", (y2 - y3).abs().max().item())\n```\n\n```text\n輸出形狀： (1, 96, 8, 8) (1, 96, 8, 8)\n兩種寫法的最大差異： 7.748603820800781e-07\n```\n\n差異只有浮點數的捨入誤差。PyTorch 的 `groups=3` 就是「把通道分成三段，各段各自卷積」，所以這個簡化的 Inception 模組，其實就是一個 $1\\times1$ 卷積加上一個分成三組的 $3\\times3$ 卷積。\n\n這樣看，簡化後的 Inception 模組只剩一個可以調的數字，也就是通道分成幾段。\n\n### 一路分到底\n\n段數越多，每段的通道越少。分到極端，每一段只剩一個通道，每個輸出通道都配一個自己的 $3\\times3$ 卷積：\n\n[![極端版本的 Inception 模組，由下往上：Input 進入一個 1x1 conv，輸出通道被分成很多段，每段只接一個 3x3，最後 Concat](/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png)](/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png)\n\n*原論文 Figure 4，取自 [**v3 PDF 第 2 頁**](https://arxiv.org/pdf/1610.02357v3#page=2)。「極端」版本的 Inception 模組，$1\\times1$ 卷積的每個輸出通道各有一個空間卷積。*\n\n我們固定 96 個通道，看段數從 1 加到 96 時，$3\\times3$ 那一層的權重怎麼變：\n\n```python\nimport torch.nn as nn\n\nC = 96\nfor groups in (1, 3, 4, 32, 96):\n    conv = nn.Conv2d(C, C, 3, padding=1, groups=groups, bias=False)\n    print(f\"groups={groups:>2}：每段 {C // groups:>2} 個通道，3×3 卷積的參數 {conv.weight.numel():>6,}，權重形狀 {tuple(conv.weight.shape)}\")\n```\n\n```text\ngroups= 1：每段 96 個通道，3×3 卷積的參數 82,944，權重形狀 (96, 96, 3, 3)\ngroups= 3：每段 32 個通道，3×3 卷積的參數 27,648，權重形狀 (96, 32, 3, 3)\ngroups= 4：每段 24 個通道，3×3 卷積的參數 20,736，權重形狀 (96, 24, 3, 3)\ngroups=32：每段  3 個通道，3×3 卷積的參數  2,592，權重形狀 (96, 3, 3, 3)\ngroups=96：每段  1 個通道，3×3 卷積的參數    864，權重形狀 (96, 1, 3, 3)\n```\n\n權重形狀的第二個數字，就是每個卷積核看得到幾個輸入通道。`groups=1` 是一般的卷積，每個卷積核看全部 96 個通道；`groups=96` 時，每個卷積核只看一個通道，只剩下空間方向的 $3\\times3$。\n\n作者把這整排看成一條光譜。一端是一般的卷積（前面接一個 $1\\times1$），只有一段；另一端是每個通道一段。Inception 模組在中間，把幾百個通道分成 3 或 4 段。\n\n### 光譜的另一端\n\n作者指出，光譜另一端的極端模組，幾乎就是已經存在的**深度可分離卷積**（depthwise separable convolution）。它分成兩步：\n\n1. **depthwise 卷積**：每個輸入通道各自做空間卷積，通道之間不交流。\n2. **pointwise 卷積**：用 $1\\times1$ 卷積把上一步的輸出投影到新的通道空間。\n\nTensorFlow、Keras 裡叫做 separable convolution 的就是它。影像處理領域也有 separable convolution 這個名字，指的是把 $7\\times7$ 拆成 $1\\times7$ 和 $7\\times1$ 這種空間上的拆法，Inception-v3 就用過，兩者不要搞混。\n\n\n> [!NOTE]\n> \n> 深度可分離卷積不是這篇發明的。作者在論文裡交代了來源，Laurent Sifre 2013 年在 Google Brain 實習時開發了它，用在 AlexNet 上，準確率小幅提升，收斂快很多，模型也小很多；GoogLeNet 和 BN-Inception 的第一層也用過。同一時間，Google 的 Andrew Howard 正在用它做行動裝置上的 MobileNets，可以看我們的 [MobileNet-V1](/blog/papers/lightweight/mobilenet-v1) 筆記。\n\n\n「極端」的 Inception 模組和深度可分離卷積，還有兩個小差別：\n\n1. **順序。** 深度可分離卷積先做 depthwise、再做 $1\\times1$；Inception 先做 $1\\times1$。作者認為這不重要，網路是一層層疊起來的，前一個模組的 $1\\times1$ 接著下一個模組的 depthwise，順序只差在頭尾。\n2. **中間有沒有非線性。** Inception 的兩步後面都接 ReLU；深度可分離卷積通常兩步之間不加激活函數。作者認為這一點可能有影響，留到實驗驗證。\n\n### Xception\n\n既然 Inception 假設是「跨通道和空間關係可以部分分開」，作者乾脆做一個更強的假設，認為兩者可以**完全**分開。基於這個假設的網路，全部用深度可分離卷積搭成，名字取自 Extreme Inception，叫做 Xception。\n\n[![Xception 的架構圖，分成三欄。Entry flow：299x299x3 影像，經過 Conv 32 3x3 stride 2、Conv 64 3x3，接著三個模組，每個是兩層 SeparableConv（128、256、728 個通道）加 MaxPooling 3x3 stride 2，旁邊是 Conv 1x1 stride 2 的捷徑，輸出 19x19x728。Middle flow：三層 ReLU 加 SeparableConv 728，外面一條恆等捷徑，重複 8 次。Exit flow：SeparableConv 728、SeparableConv 1024、MaxPooling，旁邊 Conv 1x1 stride 2 捷徑；接著 SeparableConv 1536、SeparableConv 2048、GlobalAveragePooling 得到 2048 維向量，可選的全連接層，最後 Logistic regression](/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png)](/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png)\n\n*原論文 Figure 5，取自 [**v3 PDF 第 5 頁**](https://arxiv.org/pdf/1610.02357v3#page=5)。所有 Conv 和 SeparableConv 後面都接 BN（圖上省略），SeparableConv 的 depthwise 部分不擴增通道。*\n\n資料由左往右走三段。entry flow 從 $299\\times299$ 的影像開始，前兩層是一般卷積，接著三個模組，每個模組把特徵圖縮小一半、通道加寬，最後停在 $19\\times19\\times728$。middle flow 是同一個模組重複 8 次，大小不變。exit flow 再縮小一次，把通道加到 2048，全域平均池化後接分類層。\n\n整個網路有 36 層卷積，分成 14 個模組，除了第一個和最後一個模組，每個模組外面都有一條殘差連接。不需要縮小的模組用恆等捷徑；需要縮小的，捷徑上放一個步幅 2 的 $1\\times1$ 卷積。\n\n和 Inception-v3 比，這張圖簡單很多，就是深度可分離卷積和殘差連接一層層疊起來，作者說用 Keras 寫只要 30 到 40 行。我們照圖用 PyTorch 搭一次：\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass SeparableConv(nn.Sequential):\n    \"\"\"逐通道 3×3 卷積，接著 1×1 卷積；兩者之間沒有激活函數，最後接 BN\"\"\"\n    def __init__(self, cin, cout):\n        super().__init__(\n            nn.Conv2d(cin, cin, 3, padding=1, groups=cin, bias=False),   # depthwise\n            nn.Conv2d(cin, cout, 1, bias=False),                          # pointwise\n            nn.BatchNorm2d(cout))\n\nclass Block(nn.Module):\n    def __init__(self, cin, widths, relu_first=True, downsample=True):\n        super().__init__()\n        layers, c = [], cin\n        for i, w in enumerate(widths):\n            if relu_first or i > 0:\n                layers.append(nn.ReLU())\n            layers.append(SeparableConv(c, w))\n            c = w\n        if downsample:\n            layers.append(nn.MaxPool2d(3, 2, padding=1))\n            self.shortcut = nn.Sequential(nn.Conv2d(cin, c, 1, 2, bias=False), nn.BatchNorm2d(c))\n        else:\n            self.shortcut = nn.Identity()\n        self.body = nn.Sequential(*layers)\n\n    def forward(self, x):\n        return self.body(x) + self.shortcut(x)\n\ndef conv_bn_relu(cin, cout, stride):\n    return nn.Sequential(nn.Conv2d(cin, cout, 3, stride, bias=False), nn.BatchNorm2d(cout), nn.ReLU())\n\nentry = nn.Sequential(conv_bn_relu(3, 32, 2), conv_bn_relu(32, 64, 1),\n                      Block(64, [128, 128], relu_first=False), Block(128, [256, 256]), Block(256, [728, 728]))\nmiddle = nn.Sequential(*[Block(728, [728, 728, 728], downsample=False) for _ in range(8)])\nexit_ = nn.Sequential(Block(728, [728, 1024]),\n                      SeparableConv(1024, 1536), nn.ReLU(), SeparableConv(1536, 2048), nn.ReLU(),\n                      nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(2048, 1000))\nmodel = nn.Sequential(entry, middle, exit_)\n\nx = torch.randn(1, 3, 299, 299)\nfor name, part in [(\"entry flow\", entry), (\"middle flow\", middle)]:\n    x = part(x)\n    print(f\"{name} 之後：\", tuple(x.shape[1:]))\nprint(\"exit flow 之後：\", tuple(exit_(x).shape[1:]))\n\nparams = sum(p.numel() for p in model.parameters())\nprint(f\"參數量：{params:,}\")\n```\n\n```text\nentry flow 之後： (728, 19, 19)\nmiddle flow 之後： (728, 19, 19)\nexit flow 之後： (1000,)\n參數量：22,855,952\n```\n\n參數量和論文報告的 22,855,952 完全一樣。PyTorch 沒有現成的 separable convolution 層，照這樣寫一個 `groups=輸入通道數` 的 $3\\times3$ 卷積接一個 $1\\times1$ 卷積就行；Keras 則直接有 `SeparableConv2D`，Xception 本身也收在 Keras Applications 裡。\n\n對照圖上的細節，每個 SeparableConv 前面放 ReLU、後面接 BN，depthwise 和 pointwise 之間什麼都不加。entry flow 第一個殘差模組的第一層 SeparableConv 前面沒有 ReLU，因為前面的 Conv 64 已經接了 ReLU；exit flow 最後兩層則是 ReLU 放在後面。\n\n## 討論\n\n作者拿 Inception-v3 來比，因為兩者的參數量幾乎一樣，差距不能歸因於模型容量。比較在兩個資料集上進行：\n\n- **ImageNet**：1000 類的單標籤分類。\n- **JFT**：Google 內部的資料集，超過 3.5 億張高解析度影像，17,000 個類別，每張可以有多個標籤。評估用另一個 14,000 張的 FastEval14k，平均每張 36.5 個標籤，指標是前 100 個預測的 MAP@100，並依各類別在社群媒體上的常見程度加權。\n\n兩個網路用的是同一套訓練設定，這套設定是替 Inception-v3 仔細調過的，作者沒有替 Xception 另外調。唯一的例外是 weight decay，Inception-v3 的 4e-5 對 Xception 明顯不好，改用 1e-5。所有模型都在 60 張 NVIDIA K80 上訓練，ImageNet 每次大約 3 天，JFT 每次超過一個月，而且沒有訓練到收斂。\n\n### ImageNet\n\n| 網路 | Top-1 準確率 | Top-5 準確率 |\n| --- | ---: | ---: |\n| VGG-16 | 0.715 | 0.901 |\n| ResNet-152 | 0.770 | 0.933 |\n| Inception-v3 | 0.782 | 0.941 |\n| **Xception** | **0.790** | **0.945** |\n\n*表：摘自原論文 Table 1，單一模型、單一裁切的驗證集結果。Inception-v3 沒有使用輔助分類器。*\n\nXception 比 Inception-v3 高 0.8 個百分點，差距不大。VGG-16 和 ResNet-152 只是放著參考。\n\n### JFT\n\n| 網路 | FastEval14k MAP@100 |\n| --- | ---: |\n| Inception-v3，無全連接層 | 6.36 |\n| Xception，無全連接層 | 6.70 |\n| Inception-v3，有全連接層 | 6.50 |\n| **Xception，有全連接層** | **6.78** |\n\n*表：摘自原論文 Table 2，訓練 3,000 萬次迭代（約一個月）後的結果。「有全連接層」指在分類層前加兩層 4096 維的全連接層。*\n\n在 JFT 上差距大得多，有全連接層的版本相對提升 4.3%。作者的解釋是，Inception-v3 是針對 ImageNet 發展出來的，設計上可能已經過度貼合這個資料集；JFT 則是兩個網路都沒調過。\n\n### 參數差不多，速度慢一點\n\n| 網路 | 參數量 | 每秒訓練步數 |\n| --- | ---: | ---: |\n| Inception-v3 | 23,626,728 | 31 |\n| Xception | 22,855,952 | 28 |\n\n*表：摘自原論文 Table 3，在 ImageNet 上以 60 張 K80 同步訓練。*\n\n兩者的參數量差不到 3.5%，作者因此說，Xception 的進步來自更有效率地使用參數，而不是容量變大。\n\n速度上，Xception 反而慢一點。depthwise 卷積的參數和計算量雖然少，GPU 上卻不容易跑出對應的速度。\n\n### 殘差連接\n\n作者也訓練了一個拿掉所有殘差連接的 Xception。在 ImageNet 上，沒有殘差的版本收斂慢很多，最後的準確率也明顯比較低。\n\n作者也提醒，沒有殘差的版本用的是同一套訓練設定，對它可能不太公平；而且這只說明殘差連接對**這個架構**很重要。他另外試過把 VGG 式網路的卷積全部換成深度可分離卷積、完全不用殘差，在 JFT 上一樣贏過參數量相同的 Inception-v3。\n\n### 中間要不要加激活函數\n\n回到前面留下的問題，depthwise 和 pointwise 之間要不要加非線性？作者試了不加、加 ReLU、加 ELU 三種：\n\n[![ImageNet 驗證準確率對梯度下降步數的訓練曲線。紅色的 No intermediate activation 上升最快，最後接近 0.79；綠色的 Intermediate ELU 次之，大約停在 0.78；藍色的 Intermediate ReLU 最慢，大約停在 0.77](/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png)](/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png)\n\n*原論文 Figure 10，取自 [**v3 PDF 第 7 頁**](https://arxiv.org/pdf/1610.02357v3#page=7)。在 SeparableConv 的 depthwise 與 pointwise 之間使用不同激活函數的訓練曲線。*\n\n紅線是不加任何激活函數，收斂最快，最後的準確率也最高；加 ELU 的綠線居中；加 ReLU 的藍線最差。\n\n這和 Inception-v3 論文的結果相反。Inception-v3 把 $5\\times5$ 分解成兩層 $3\\times3$ 時，發現中間加 ReLU 比較好。作者的推測是，關鍵在於空間卷積所在的特徵空間有多「厚」。Inception 模組裡，每段有幾十上百個通道，非線性有幫助；深度可分離卷積的每段只有一個通道，ReLU 把負值砍掉，資訊可能就這樣丟了。\n\n後來的 MobileNetV2 在網路中最窄的那幾層拿掉 ReLU，用的也是「低維空間做非線性會丟資訊」的理由，可以接著看 [MobileNet-V2](/blog/papers/lightweight/mobilenet-v2)。\n\n## 結論\n\nXception 換了一個角度看 Inception。簡化之後的 Inception 模組，只是一個 $1\\times1$ 卷積加上分成幾組的 $3\\times3$ 卷積；組數從 1 調到通道數，就從一般卷積走到深度可分離卷積。Xception 直接走到盡頭，參數量和 Inception-v3 幾乎相同，ImageNet 上略好，JFT 上好得比較多。\n\n名字叫 Extreme Inception，速度倒是一點也不極端，每秒訓練步數還比 Inception-v3 少 3 步。作者當時預期，depthwise 卷積的實作優化之後就會反超；這一等就是好幾年，[EfficientNetV2](/blog/papers/classic-cnns/efficientnet-v2) 最後乾脆把網路前段的 depthwise 卷積換掉了。\n\n至於盡頭是不是最好的位置，作者自己也不確定，他在文末寫道，光譜中間的某些點可能更好。一個多月後的 [ResNeXt](/blog/papers/classic-cnns/resnext) 用了分成 32 組的卷積，剛好落在中間。\n","imageSizes":{"/content/papers/assets/9fc3c2cfbe884067fa627f67d745ee0ab6eaf44ed451260083932eb5fa0742e0.png":{"width":917,"height":554},"/content/papers/assets/b56e8b540dd5a4cb9856460b3e47a6e3e624df308ab08b70809438075caf5f0a.png":{"width":917,"height":538},"/content/papers/assets/8b7417d8bf8ba80d77cf08ea108b8a7df9b994d3f4f08922532004a3ce0f7f69.png":{"width":2054,"height":1417},"/content/papers/assets/c1da1e28d13c72c20287fa73c73e3e10c9dc4f735991c6f580191ede66fd1b5b.png":{"width":946,"height":716}}}