{"notes":[{"id":"classic-cnns/inception-v4","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v4","subtitle":"快很多，準一點","author":"Z. Yuan","paperDate":"2016-02-23","publishedOn":"2026-01-27","world":"external/inception-v4","paperTitle":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","paperUrl":"https://arxiv.org/abs/1602.07261","bibliography":{"title":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning","authors":["Christian Szegedy","Sergey Ioffe","Vincent Vanhoucke","Alex Alemi"],"url":"https://arxiv.org/abs/1602.07261","pdfUrl":"https://arxiv.org/pdf/1602.07261v2","codeUrl":null,"preprint":{"id":"1602.07261","submittedOn":"2016-02-23","url":"https://arxiv.org/abs/1602.07261"},"publication":null,"citation":"@misc{classiccnnsinceptionv42016,\n  title = {{Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning}},\n  author = {Christian Szegedy and Sergey Ioffe and Vincent Vanhoucke and Alex Alemi},\n  year = {2016},\n  eprint = {1602.07261},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1602.07261}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v4.ed8e0294ae0b.bib"},"thumbnail":{"src":"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png","width":666,"height":656},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/inception-v4.a69e9457230b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"卸下包袱inception-v4","text":"卸下包袱：Inception-v4","depth":3},{"id":"進場的那一段","text":"進場的那一段","depth":3},{"id":"把殘差接進-inception","text":"把殘差接進 Inception","depth":3},{"id":"縮小殘差穩住訓練","text":"縮小殘差，穩住訓練","depth":3},{"id":"訓練設定","text":"訓練設定","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"殘差讓訓練快很多","text":"殘差讓訓練快很多","depth":3},{"id":"單一模型的成績","text":"單一模型的成績","depth":3},{"id":"集成","text":"集成","depth":3},{"id":"殘差連接是必需的嗎","text":"殘差連接是「必需」的嗎？","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""},{"id":"classic-cnns/resnet-v2","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"卷積網路 100 層","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/resnet.859f62e9e37d.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"什麼是殘差","text":"什麼是殘差？","depth":3},{"id":"殘差學習","text":"殘差學習","depth":3},{"id":"殘差模塊","text":"殘差模塊","depth":3},{"id":"網路架構","text":"網路架構","depth":3},{"id":"參數配置","text":"參數配置","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"更深的瓶頸架構","text":"更深的瓶頸架構","depth":3},{"id":"在-imagenet-上的表現","text":"在 ImageNet 上的表現","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"如何訓練更深的卷積網路"},{"id":"classic-cnns/densenet","relation":"next","topic":"如何訓練更深的卷積網路"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"大卷積核退場","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2025-12-16","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":null,"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/inception-v3.b02feaa71206.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"把-55-拆成兩層-33","text":"把 5×5 拆成兩層 3×3","depth":3},{"id":"再拆成-1n-與-n1","text":"再拆成 1×n 與 n×1","depth":3},{"id":"輔助分類器的真正作用","text":"輔助分類器的真正作用","depth":3},{"id":"縮小特徵圖又不想卡住資訊","text":"縮小特徵圖，又不想卡住資訊","depth":3},{"id":"把零件組起來","text":"把零件組起來","depth":3},{"id":"讓模型別太有把握","text":"讓模型別太有把握","depth":3},{"id":"訓練設定","text":"訓練設定","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"一項一項加上去","text":"一項一項加上去","depth":3},{"id":"解析度低一點差多少","text":"解析度低一點，差多少？","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ResNet v2","subtitle":"捷徑要乾淨","author":"Z. Yuan","paperDate":"2016-03-16","publishedOn":"2026-02-14","world":"external/resnet-v2","paperTitle":"Identity Mappings in Deep Residual Networks","paperUrl":"https://arxiv.org/abs/1603.05027","bibliography":{"title":"Identity Mappings in Deep Residual Networks","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1603.05027","pdfUrl":"https://arxiv.org/pdf/1603.05027v3","codeUrl":null,"preprint":{"id":"1603.05027","submittedOn":"2016-03-16","url":"https://arxiv.org/abs/1603.05027"},"publication":null,"citation":"@misc{classiccnnsresnetv22016,\n  title = {{Identity Mappings in Deep Residual Networks}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2016},\n  eprint = {1603.05027},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.05027}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet-v2.3d311a21e308.bib"},"thumbnail":{"src":"/content/papers/assets/7eda4acdde3019e92c5b44f59917b1e493a5b11b5b8a63280d03b15787934ef5.png","width":967,"height":469},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/resnet-v2.63e5bb8d2aec.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"兩個恆等映射","text":"兩個恆等映射","depth":3},{"id":"捷徑上多一點東西會怎樣","text":"捷徑上多一點東西會怎樣？","depth":3},{"id":"相加之後的-relu-該放哪","text":"相加之後的 ReLU 該放哪？","depth":3},{"id":"換個角度看pre-activation","text":"換個角度看：pre-activation","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"越深差距越大","text":"越深，差距越大","depth":3},{"id":"和其他方法比","text":"和其他方法比","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""},{"id":"classic-cnns/highway-networks","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v4","relation":"referenced","topic":""}]}],"text":"ILSVRC 2015 的分類冠軍是微軟研究院的 ResNet。它的架構很傳統，一層層卷積疊上去，只多了殘差連接；單一模型的成績，和 Google 剛發表的 Inception-v3 差不多。\n\n兩條路線各有一套做法：Inception 靠多分支的模組和精打細算的通道數，把計算量壓低；ResNet 靠殘差連接，讓很深的網路也訓練得動。\n\n那兩個合在一起呢？\n\n2016 年 2 月，Google 的 Christian Szegedy、Sergey Ioffe、Vincent Vanhoucke 與 Alex Alemi 放出這篇論文，後來發表在 AAAI 2017。標題把三件事都寫進去了：新的 Inception-v4、加上殘差連接的 Inception-ResNet，以及殘差連接對訓練到底有什麼影響。\n\n\n> [!TIP]\n> \n> 這篇的兩個前作，可以先看我們的筆記：\n> \n> - **[[15.12] ResNet：卷積網路 100 層](/blog/papers/classic-cnns/resnet)**\n> - **[[15.12] Inception-v3：大卷積核退場](/blog/papers/classic-cnns/inception-v3)**\n\n\n## 定義問題\n\n這篇論文同時處理兩個問題。\n\n第一個是殘差連接。ResNet 的作者主張，殘差連接是訓練非常深的卷積網路**本質上需要**的東西。Inception 網路本來就很深，把 Inception 模組最後的「通道接起來」換成「加回輸入」，看起來很自然。問題是：這樣做有什麼好處？好處是在最後的準確率，還是在別的地方？\n\n第二個是 Inception 自己。作者坦白說，Inception-v3 背了很多歷史包袱。早期的模型用 DistBelief 分散式訓練，每個副本要切成好幾個子網路才放得進記憶體；為了讓各個子網路的計算量平衡，各層的濾波器數量都經過仔細調整。換到 TensorFlow 之後，這個限制沒有了。另一方面，他們過去改架構一向保守，一次只動一個元件，其餘不變，結果網路越長越複雜。\n\n既然包袱已經可以放下，Inception 能不能變得更簡單、更深、更寬？\n\n## 解決問題\n\n### 卸下包袱：Inception-v4\n\n[![Inception-v4 的整體架構，由下往上：Input 299x299x3、Stem（輸出 35x35x384）、4 x Inception-A（35x35x384）、Reduction-A（17x17x1024）、7 x Inception-B（17x17x1024）、Reduction-B（8x8x1536）、3 x Inception-C（8x8x1536）、Average Pooling（1536）、Dropout（keep 0.8）、Softmax（1000）](/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png)](/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png)\n\n*原論文 Figure 9，取自 [**v2 PDF 第 5 頁**](https://arxiv.org/pdf/1602.07261v2#page=5)。Inception-v4 的整體架構，右側是每一段的輸出大小。*\n\n這張圖和 Inception-v3 的架構表很像：三種解析度（$35\\times35$、$17\\times17$、$8\\times8$），每一段疊幾個 Inception 模組，中間用縮小模組換到下一個解析度。差別在於 Inception-v4 **同一個解析度只用一種模組**，數量也變多了：4 個 Inception-A、7 個 Inception-B、3 個 Inception-C。\n\n三種模組延續 Inception-v3 的分解方法：\n\n- **Inception-A**（$35\\times35$）：四條分支，最長的一條用兩層 $3\\times3$ 取代 $5\\times5$，每條輸出 96 個通道，合起來 384。\n- **Inception-B**（$17\\times17$）：用 $1\\times7$ 和 $7\\times1$ 的不對稱卷積，合起來 1024 個通道。\n- **Inception-C**（$8\\times8$）：把 $1\\times3$ 和 $3\\times1$ 並排輸出，合起來 1536 個通道。\n\n圖上標了「V」的卷積是 valid padding，不補零，輸出會變小；沒有標的都是 same padding，輸出大小和輸入一樣。\n\n### 進場的那一段\n\n和 Inception-v3 比，最明顯的改變在網路的開頭。作者把它叫做 stem：\n\n[![Inception-v4 的 stem，由下往上：Input 299x299x3；3x3 Conv（32 stride 2 V）得到 149x149x32；3x3 Conv（32 V）得到 147x147x32；3x3 Conv（64）得到 147x147x64；接著分成 3x3 MaxPool（stride 2 V）與 3x3 Conv（96 stride 2 V）兩條，Filter concat 得到 73x73x160；再分成兩條：1x1 Conv（64）接 3x3 Conv（96 V），以及 1x1 Conv（64）、7x1 Conv（64）、1x7 Conv（64）、3x3 Conv（96 V），Filter concat 得到 71x71x192；最後分成 3x3 Conv（192 V）與 MaxPool（stride=2 V）兩條，Filter concat 得到 35x35x384](/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png)](/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png)\n\n*原論文 Figure 3，取自 [**v2 PDF 第 3 頁**](https://arxiv.org/pdf/1602.07261v2#page=3)。Inception-v4 與 Inception-ResNet-v2 共用的 stem。*\n\n前三層和 Inception-v3 一樣，從 $299\\times299$ 走到 $147\\times147\\times64$。之後連續三次「分開走、再接起來」，每一次都有一條分支負責縮小，或是兩條分支各自處理再合併。我們照圖走一次：\n\n```python\nimport torch\nimport torch.nn as nn\n\ndef conv(cin, cout, k, stride=1, valid=False):\n    # 標了 V 的卷積不補零；其他卷積補零，讓輸出大小和輸入相同\n    pad = 0 if valid else (k[0] // 2, k[1] // 2) if isinstance(k, tuple) else k // 2\n    return nn.Sequential(nn.Conv2d(cin, cout, k, stride, pad), nn.ReLU())\n\nx = torch.randn(1, 3, 299, 299)\n\nx = conv(3, 32, 3, stride=2, valid=True)(x)\nx = conv(32, 32, 3, valid=True)(x)\nx = conv(32, 64, 3)(x)\nprint(\"三層卷積後：\", tuple(x.shape[1:]))\n\n# 第一次並排：最大池化 | 步幅 2 的卷積\na = nn.MaxPool2d(3, 2)(x)\nb = conv(64, 96, 3, stride=2, valid=True)(x)\nx = torch.cat([a, b], dim=1)\nprint(\"池化\", tuple(a.shape[1:]), \"+ 卷積\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n\n# 第二次並排：兩條卷積分支\na = nn.Sequential(conv(160, 64, 1), conv(64, 96, 3, valid=True))(x)\nb = nn.Sequential(conv(160, 64, 1), conv(64, 64, (7, 1)), conv(64, 64, (1, 7)), conv(64, 96, 3, valid=True))(x)\nx = torch.cat([a, b], dim=1)\nprint(\"分支一\", tuple(a.shape[1:]), \"+ 分支二\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n\n# 第三次並排：步幅 2 的卷積 | 最大池化\na = conv(192, 192, 3, stride=2, valid=True)(x)    # 要和池化的 35×35 接上，步幅必須是 2\nb = nn.MaxPool2d(3, 2)(x)\nx = torch.cat([a, b], dim=1)\nprint(\"卷積\", tuple(a.shape[1:]), \"+ 池化\", tuple(b.shape[1:]), \"→\", tuple(x.shape[1:]))\n```\n\n```text\n三層卷積後： (64, 147, 147)\n池化 (64, 73, 73) + 卷積 (96, 73, 73) → (160, 73, 73)\n分支一 (96, 71, 71) + 分支二 (96, 71, 71) → (192, 71, 71)\n卷積 (192, 35, 35) + 池化 (192, 35, 35) → (384, 35, 35)\n```\n\n輸出的形狀和圖上標的 $73\\times73\\times160$、$71\\times71\\times192$、$35\\times35\\times384$ 一致。池化不改變通道數，所以第一次合併是 $64+96=160$，最後一次是 $192+192=384$。\n\n第一次和第三次合併，就是 Inception-v3 縮小特徵圖的方法：步幅 2 的卷積和步幅 2 的池化並排，避免先池化造成的資訊瓶頸。這次連網路的開頭也這樣做了。\n\n### 把殘差接進 Inception\n\n殘差版本的 Inception，作者用的是比較便宜的模組。以 Inception-ResNet-v2 的 $35\\times35$ 模組為例：\n\n[![Inception-ResNet-A 模組：下方的 Relu activation 分成四條路。最左邊一條直接連到上方的加號；另外三條分支分別是 1x1 Conv（32）、1x1 Conv（32）接 3x3 Conv（32）、1x1 Conv（32）接 3x3 Conv（48）再接 3x3 Conv（64）。三條分支的輸出一起進入 1x1 Conv（384 Linear），再和最左邊的直通路徑相加，最後經過 Relu activation](/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png)](/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png)\n\n*原論文 Figure 16，取自 [**v2 PDF 第 8 頁**](https://arxiv.org/pdf/1602.07261v2#page=8)。Inception-ResNet-v2 的 $35\\times35$ 模組（Inception-ResNet-A）。*\n\n和純 Inception 模組相比，有兩個地方不一樣：\n\n1. 分支比較窄。三條分支最後只有 $32+32+64=128$ 個通道，遠少於輸入的 384。\n2. 分支接起來之後，多了一層**沒有激活函數**的 $1\\times1$ 卷積，把通道數擴回 384，才能和輸入相加。作者把它叫做 filter-expansion layer，用來補回 Inception 模組造成的降維。\n\n```python\nimport torch\nimport torch.nn as nn\n\ndef conv(cin, cout, k):\n    return nn.Sequential(nn.Conv2d(cin, cout, k, padding=k // 2, bias=False), nn.BatchNorm2d(cout), nn.ReLU())\n\nclass InceptionResNetA(nn.Module):\n    def __init__(self, channels=384, scale=0.1):\n        super().__init__()\n        self.b1 = conv(channels, 32, 1)\n        self.b2 = nn.Sequential(conv(channels, 32, 1), conv(32, 32, 3))\n        self.b3 = nn.Sequential(conv(channels, 32, 1), conv(32, 48, 3), conv(48, 64, 3))\n        self.expand = nn.Conv2d(32 + 32 + 64, channels, 1)   # 1×1，沒有激活函數\n        self.scale = scale\n\n    def forward(self, x):\n        branches = torch.cat([self.b1(x), self.b2(x), self.b3(x)], dim=1)\n        residual = self.expand(branches)\n        print(\"分支接起來：\", tuple(branches.shape[1:]), \"→ 1×1 擴回：\", tuple(residual.shape[1:]))\n        return torch.relu(x + self.scale * residual)\n\nx = torch.randn(2, 384, 35, 35)\nprint(\"輸出：\", tuple(InceptionResNetA()(x).shape[1:]))\n```\n\n```text\n分支接起來： (128, 35, 35) → 1×1 擴回： (384, 35, 35)\n輸出： (384, 35, 35)\n```\n\n程式裡的 `scale` 是下一節的主角，先留著。\n\n作者試了好幾種殘差版本，論文只詳細列出兩個：\n\n| 網路 | 計算量對照 | Stem | 主體 |\n| --- | --- | --- | --- |\n| Inception-ResNet-v1 | 約等於 Inception-v3 | 自己的 stem（Figure 14） | 5 × A、Reduction-A、10 × B、Reduction-B、5 × C |\n| Inception-ResNet-v2 | 約等於 Inception-v4 | 和 Inception-v4 相同（Figure 3） | 同上，模組較寬 |\n\n*表：整理自原論文第 3.2 節與 Figure 15。兩個網路的整體結構相同，模組的通道數不同。*\n\n雖然 Inception-ResNet-v2 和 Inception-v4 的原始計算量相當，作者也提到，Inception-v4 實際上每一步訓練慢很多，可能是因為層數比較多。\n\n還有一個細節：Inception-ResNet 的 batch normalization 只加在一般的層上，**相加之後不加**。作者說，全面使用 batch normalization 應該會更好，但激活值很大的那些層，batch normalization 會吃掉不成比例的 GPU 記憶體；為了讓每個模型副本放得進單張 GPU，他們拿掉了這些 BN，換來可以多疊很多模組。\n\n### 縮小殘差，穩住訓練\n\n接下來，作者遇到了一個麻煩：濾波器數量超過 1000 時，殘差版本變得不穩定，網路會在訓練初期「死掉」。具體的症狀是，訓練幾萬次迭代之後，平均池化前的最後一層只輸出 0。調低學習率、在這一層多加一個 batch normalization，都救不回來。\n\n有效的做法很簡單：在相加之前，先把殘差乘上一個小的常數。\n\n[![殘差縮放的示意圖：下方的 Relu activation 分成兩條路，一條直接連到上方的加號，另一條經過 Inception 區塊，再經過 Activation Scaling 區塊，才連到加號；相加之後是 Relu activation](/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png)](/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png)\n\n*原論文 Figure 20，取自 [**v2 PDF 第 9 頁**](https://arxiv.org/pdf/1602.07261v2#page=9)。在 Inception-ResNet 模組裡縮放殘差，常數通常取 0.1 左右。*\n\n作者一般選 0.1 到 0.3 之間的值。ResNet 論文遇到很深的網路不穩定時，用的是先以很低的學習率「暖身」、再提高學習率的兩階段訓練；作者發現濾波器很多時，連 0.00001 的學習率都擋不住不穩定，之後的高學習率還可能把暖身的效果毀掉。相比之下，縮放殘差可靠得多。就算原本不縮放也能訓練，加上縮放也沒有傷害最終準確率，訓練還更穩。\n\n論文沒有分析為什麼會「死掉」。不過有一件事很容易看到：Inception-ResNet 的分支裡有 batch normalization，輸出的大小大致固定；主幹的相加之後沒有 batch normalization，每經過一個模組就累積一次。我們用一個簡化的模型看主幹上的訊號怎麼變：\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nx0 = torch.randn(4096, 256)                     # 主幹上的訊號：4096 個位置，256 個通道\n\nfor scale in (1.0, 0.3, 0.1):\n    x = x0.clone()\n    stds = []\n    for block in range(1, 41):\n        w = torch.randn(256, 256) / 16\n        r = torch.relu(x) @ w\n        r = (r - r.mean(0)) / r.std(0)          # 分支裡有 BN：每個通道的輸出被拉回標準差 1\n        x = x + scale * r                       # 主幹的相加之後沒有 BN\n        if block in (10, 20, 40):\n            stds.append(f\"{x.std().item():5.2f}\")\n    print(f\"縮放 {scale}：第 10、20、40 個區塊後，主幹的標準差\", \" \".join(stds))\n```\n\n```text\n縮放 1.0：第 10、20、40 個區塊後，主幹的標準差  3.31  4.57  6.42\n縮放 0.3：第 10、20、40 個區塊後，主幹的標準差  1.37  1.67  2.15\n縮放 0.1：第 10、20、40 個區塊後，主幹的標準差  1.05  1.10  1.18\n```\n\n不縮放時，每個模組都加進一份和主幹差不多大的訊號，40 個模組之後，標準差大約是 $\\sqrt{41}\\approx6.4$。縮放 0.1 時，每個模組只做一點小修正，主幹上的訊號主要還是從直通路徑傳下來的。這只是初始化時的簡化示意，不能解釋訓練中途的「死掉」；論文能確定的只有實驗結果：縮放之後，寬的殘差網路訓練得穩。\n\n### 訓練設定\n\n和 Inception-v3 幾乎一樣：TensorFlow 分散式訓練，這次用 20 個副本，各跑在一張 NVIDIA Kepler GPU 上。最好的模型用 RMSProp（decay 0.9、$\\epsilon=1.0$），學習率 0.045，每兩個 epoch 乘上 0.94；評估時使用參數的移動平均。\n\n## 討論\n\n### 殘差讓訓練快很多\n\n[![折線圖，橫軸是 epoch（20 到 200），縱軸是 top-1 錯誤率（15% 到 29%）。綠色虛線是 inception-v3，紅色實線是 inception-resnet-v1。紅線下降得快很多，大約 80 個 epoch 就接近 22%，綠線要到 160 個 epoch 左右才追上；最後兩條線都停在 21% 多一點](/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png)](/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png)\n\n*原論文 Figure 21，取自 [**v2 PDF 第 9 頁**](https://arxiv.org/pdf/1602.07261v2#page=9)。計算量相近的 Inception-v3 與 Inception-ResNet-v1，訓練過程中的 top-1 錯誤率（單一裁切）。*\n\n紅線是加了殘差的 Inception-ResNet-v1，綠線是 Inception-v3。紅線下降得快很多，可是兩條線最後停在差不多的地方：top-1 是殘差版本略差一點，top-5 則是殘差版本略好一點。Inception-v4 與 Inception-ResNet-v2 這一對，也是殘差版本訓練得比較快，最後的成績略好。\n\n四個模型放在一起比，作者的結論是：殘差版本收斂得比較快，但最終準確率主要取決於模型大小。\n\n### 單一模型的成績\n\n| 網路 | Top-1 錯誤率 | Top-5 錯誤率 |\n| --- | ---: | ---: |\n| BN-Inception | 25.2% | 7.8% |\n| Inception-v3 | 21.2% | 5.6% |\n| Inception-ResNet-v1 | 21.3% | 5.5% |\n| Inception-v4 | 20.0% | 5.0% |\n| **Inception-ResNet-v2** | **19.9%** | **4.9%** |\n\n*表：摘自原論文 Table 2。單一模型、單一裁切，驗證集排除黑名單影像。*\n\n計算量相近的兩兩一組來看：Inception-v3 和 Inception-ResNet-v1 幾乎打平，Inception-v4 和 Inception-ResNet-v2 也只差 0.1 個百分點。兩組之間的差距，比組內有沒有殘差的差距大得多。\n\n多裁切的結果也是一樣的走勢：\n\n| 網路 | 裁切數 | Top-1 錯誤率 | Top-5 錯誤率 |\n| --- | ---: | ---: | ---: |\n| ResNet-152 | 10 | 21.4% | 5.7% |\n| Inception-v3 | 12 | 19.8% | 4.6% |\n| Inception-ResNet-v1 | 12 | 19.8% | 4.6% |\n| Inception-v4 | 12 | 18.7% | 4.2% |\n| Inception-ResNet-v2 | 12 | 18.7% | 4.1% |\n| ResNet-152 | dense | 19.4% | 4.5% |\n| Inception-v3 | 144 | 18.9% | 4.3% |\n| Inception-ResNet-v1 | 144 | 18.8% | 4.3% |\n| Inception-v4 | 144 | 17.7% | 3.8% |\n| Inception-ResNet-v2 | 144 | 17.8% | 3.7% |\n\n*表：摘自原論文 Table 3 與 Table 4，完整的 50,000 張驗證集。*\n\n### 集成\n\n| 網路 | 模型數 | Top-1 錯誤率 | Top-5 錯誤率 |\n| --- | ---: | ---: | ---: |\n| ResNet-152 | 6 | – | 3.6% |\n| Inception-v3 | 4 | 17.3% | 3.6% |\n| **Inception-v4 + 3 × Inception-ResNet-v2** | **4** | **16.5%** | **3.1%** |\n\n*表：摘自原論文 Table 5，完整的 50,000 張驗證集。ResNet 為 dense 評估，Inception 系列為 144 個裁切。*\n\n一個 Inception-v4 加三個 Inception-ResNet-v2，驗證集 top-5 錯誤率 3.1%。這組集成也送到測試伺服器，得到 3.08%。作者特別說明，過去一年他們只向測試伺服器送過兩次結果，一次是 BN-Inception 論文，一次是 ILSVRC 2015 的 CLSLOC 比賽，所以測試集的數字可以看成對泛化能力的真實估計。\n\n作者也注意到，單一模型的進步，到了集成就沒有那麼明顯。\n\n### 殘差連接是「必需」的嗎？\n\n回到開頭的第一個問題。作者的回答很直接：至少在影像辨識上，他們的結果不支持 ResNet 論文「殘差連接本質上必需」的說法。Inception-v4 沒有殘差連接，一樣訓練得起來，成績也和 Inception-ResNet-v2 相當。\n\n他們也保留了餘地：要看清殘差連接的完整好處，可能需要更深的架構、更多的實驗點。能確定的是，殘差連接讓訓練快很多，光這一點就值得使用。\n\n## 結論\n\n這篇論文交出了三個網路：純 Inception 的 Inception-v4，以及計算量分別對齊 Inception-v3、Inception-v4 的 Inception-ResNet-v1、v2。\n\n比較的條件要先說清楚：Inception-ResNet 的模組是「挑得有點隨意」的（作者原話是 somewhat ad hoc），只要求計算量和對應的純 Inception 模型相近；作者也試過更大、更寬的版本，在 ImageNet 上的表現很接近。在這個條件下，殘差連接帶來的主要是訓練速度；真正拉開差距的是模型變大，Inception-v4 和 Inception-ResNet-v2 都明顯好過 v3 那一級的模型。\n\n這篇的結論對拿模型來用的人很實際：在意訓練時間，就加上殘差連接；在意最後的準確率，模型大小比有沒有殘差重要。網路很寬時，記得先把殘差乘上 0.1 到 0.3 再加回去，不然可能訓練到一半就「死掉」。\n\n至於殘差是不是訓練深網路的必需品，ResNet 和這篇各有說法。同一時期，ResNet 的作者自己也在研究殘差該怎麼接才對，就是下一篇的 [ResNet v2](/blog/papers/classic-cnns/resnet-v2)。\n","imageSizes":{"/content/papers/assets/f491d08afae7d54b8ffcbd407c61918990579e762722f0424cf7813284d73873.png":{"width":335,"height":941},"/content/papers/assets/a8aa67238a41bcf306ee4cf1d7d45c7564e9b522e9f9ea4272cbda8e3c2787e8.png":{"width":431,"height":1189},"/content/papers/assets/cc8335c4ba4672514d3dd5e9953719d52b4f8fdd0af994c76722874543fcb864.png":{"width":666,"height":656},"/content/papers/assets/80acad99318e473fe9dab6f2b4d92977d9619ab66149982bd22ff1e8067a7917.png":{"width":277,"height":396},"/content/papers/assets/2c7a11bd8741d531c7a4888f1c2efee55cf3b7f27c1fb808e2f1da765a585065.png":{"width":565,"height":356}}}