{"notes":[{"id":"classic-cnns/inceptionnext","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InceptionNeXt","subtitle":"拆成十字","author":"Z. Yuan","paperDate":"2023-03-29","publishedOn":"2025-10-30","world":"external/inceptionnext","paperTitle":"InceptionNeXt: When Inception Meets ConvNeXt","paperUrl":"https://arxiv.org/abs/2303.16900","bibliography":{"title":"InceptionNeXt: When Inception Meets ConvNeXt","authors":["Weihao Yu","Pan Zhou","Shuicheng Yan","Xinchao Wang"],"url":"https://arxiv.org/abs/2303.16900","pdfUrl":"https://arxiv.org/pdf/2303.16900v3","codeUrl":"https://github.com/sail-sg/inceptionnext","preprint":{"id":"2303.16900","submittedOn":"2023-03-29","url":"https://arxiv.org/abs/2303.16900"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Yu_InceptionNeXt_When_Inception_Meets_ConvNeXt_CVPR_2024_paper.html"},"citation":"@misc{classiccnnsinceptionnext2023,\n  title = {{InceptionNeXt: When Inception Meets ConvNeXt}},\n  author = {Weihao Yu and Pan Zhou and Shuicheng Yan and Xinchao Wang},\n  year = {2023},\n  eprint = {2303.16900},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2303.16900}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inceptionnext.82116f5daa6e.bib"},"thumbnail":{"src":"/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png","width":1446,"height":786},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/inceptionnext.232785d24ca0.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"metanext","text":"MetaNeXt","depth":3},{"id":"慢在哪裡","text":"慢在哪裡","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"inception-depthwise-卷積","text":"Inception depthwise 卷積","depth":3},{"id":"十字形的感受野","text":"十字形的感受野","depth":3},{"id":"組成網路","text":"組成網路","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-1k","text":"ImageNet-1K","depth":3},{"id":"metanext-裝不進注意力","text":"MetaNeXt 裝不進注意力","depth":3},{"id":"語意分割","text":"語意分割","depth":3},{"id":"消融","text":"消融","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"vision-transformers/poolformer","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"讓卷積再次偉大","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/convnext.667fd92d121a.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"重新設計網路架構","text":"重新設計網路架構","depth":3},{"id":"引入現代訓練技術","text":"引入現代訓練技術","depth":3},{"id":"宏觀設計","text":"宏觀設計","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"大核卷積","text":"大核卷積","depth":3},{"id":"微觀設計","text":"微觀設計","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"總結模型配置","text":"總結模型配置","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"用卷積混合影像切塊"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"用卷積混合影像切塊"},{"id":"reparameterization/replknet","relation":"comparison","topic":"大卷積核與簡化骨架"}]},{"id":"vision-transformers/poolformer","category":"vision-transformers","legacyPath":"vision-transformers/2111-poolformer","legacySlug":"2111-poolformer","legacyPaths":[],"title":"PoolFormer","subtitle":"你需要的是框架！","author":"Z. Yuan","paperDate":"2021-11-22","publishedOn":"2024-07-10","paperTitle":"MetaFormer Is Actually What You Need for Vision","paperUrl":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","bibliography":{"title":"MetaFormer Is Actually What You Need for Vision","authors":["Weihao Yu","Mi Luo","Pan Zhou","Chenyang Si","Yichen Zhou","Xinchao Wang","Jiashi Feng","Shuicheng Yan"],"url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2022/papers/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.pdf","codeUrl":"https://github.com/sail-sg/poolformer","preprint":{"id":"2111.11418","submittedOn":"2021-11-22","url":"https://arxiv.org/abs/2111.11418"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html"},"citation":"@misc{visiontransformerspoolformer2021,\n  title = {{MetaFormer Is Actually What You Need for Vision}},\n  author = {Weihao Yu and Mi Luo and Pan Zhou and Chenyang Si and Yichen Zhou and Xinchao Wang and Jiashi Feng and Shuicheng Yan},\n  year = {2021},\n  eprint = {2111.11418},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.11418}\n}\n","citationPath":"/content/papers/citations/vision-transformers/poolformer.f2f96629d283.bib"},"thumbnail":{"src":"/content/papers/assets/3cb141e9bb1247410c8214276b21b28ed1d536060d470406f1bfb0c88fe68df2.jpg","width":1498,"height":548},"locale":"zh-hant","path":"/content/papers/zh-hant/vision-transformers/poolformer.cc4d5c111221.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"input-embedding","text":"Input Embedding","depth":3},{"id":"token-mixer","text":"Token-Mixer","depth":3},{"id":"channel-mlp","text":"Channel MLP","depth":3},{"id":"或許-token-mixer-不是重點","text":"或許 Token-Mixer 不是重點？","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"模型配置","text":"模型配置","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-實驗","text":"ImageNet 實驗","depth":3},{"id":"消融實驗","text":"消融實驗","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/mlp-mixer","relation":"background","topic":"把骨架與 Token 混合方式分開看"},{"id":"vision-transformers/caformer","relation":"next","topic":"把骨架與 Token 混合方式分開看"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"大卷積核退場","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2023-11-15","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html"},"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/inception-v3.fb3811ac4302.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"把-55-拆成兩層-33","text":"把 5×5 拆成兩層 3×3","depth":3},{"id":"再拆成-1n-與-n1","text":"再拆成 1×n 與 n×1","depth":3},{"id":"輔助分類器的真正作用","text":"輔助分類器的真正作用","depth":3},{"id":"縮小特徵圖又不想卡住資訊","text":"縮小特徵圖，又不想卡住資訊","depth":3},{"id":"把零件組起來","text":"把零件組起來","depth":3},{"id":"讓模型別太有把握","text":"讓模型別太有把握","depth":3},{"id":"訓練設定","text":"訓練設定","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"一項一項加上去","text":"一項一項加上去","depth":3},{"id":"解析度低一點差多少","text":"解析度低一點，差多少？","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]}],"text":"ConvNeXt-T 的計算量是 4.5 GMACs，ResNet-50 是 4.1 GMACs，兩者差不多。放到 A100 上訓練，ResNet-50 每秒可以處理 969 張圖，ConvNeXt-T 只有 575 張，大約六成。\n\n計算量差不多，速度卻差這麼多，問題出在 7×7 的 depthwise 卷積。它每個通道各算各的，乘加次數很少，要搬的資料卻不少，GPU 大部分時間都在等記憶體。ConvNeXt 把卷積核從 3×3 放大到 7×7，換來了準確率，也換來這個速度問題。\n\n這篇來自新加坡國立大學、新加坡管理大學、Sea AI Lab 和 Skywork AI。第一作者 Weihao Yu 也是 MetaFormer 的作者，當時他把 Transformer 拆成「token mixer 加 MLP」的框架；這次同樣的拆法用在 ConvNeXt 上。\n\n\n> [!TIP]\n> \n> ConvNeXt 和 MetaFormer 可以先看這兩篇：\n> \n> - **[[22.01] ConvNeXt：讓卷積再次偉大](/blog/papers/classic-cnns/convnext)**\n> - **[[21.11] PoolFormer：你需要的是框架！](/blog/papers/vision-transformers/poolformer)**\n\n\n[![原論文 Figure 1：橫軸是 A100 上的訓練吞吐量（張／秒），縱軸是 ImageNet top-1 準確率。ConvNeXt-T/k7 在左上方約 82.1%、575 張／秒；換成 5×5 和 3×3 卷積核後往右移，準確率逐步下降；ResNet-50 在右下角，速度約 970 張／秒但準確率只有 78% 多；InceptionNeXt-T 在右上方，紅色箭頭標示比 ConvNeXt-T/k7 快 1.6 倍。](/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png)](/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png)\n\n*原論文 Figure 1（Yu 等人），取自 [**PDF 第 1 頁**](https://arxiv.org/pdf/2303.16900v3#page=1)。圖說未收錄。*\n\n圖的右上角是這篇要的位置，準確率不輸 ConvNeXt-T，速度接近 ResNet-50。\n\n## 定義問題\n\n### MetaNeXt\n\n作者先把 ConvNeXt 的區塊抽象化。\n\n[![原論文 Figure 2：四個區塊由左到右排列。MetaFormer 區塊有兩條殘差連接，第一條包住 Norm 和虛線框的 Token Mixer，第二條包住 Norm 和 MLP；MetaNeXt 區塊只有一條殘差連接，Token Mixer、Norm、MLP 串在同一條路上；ConvNeXt 區塊把 Token Mixer 換成 7×7 的 DWConv；InceptionNeXt 區塊的 Token Mixer 是一個虛線框，輸入先 Split 成四份，分別經過 3×3、1×11、11×1 的 DWConv 和 Identity，再 Concat 起來。](/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png)](/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png)\n\n*原論文 Figure 2（Yu 等人），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/2303.16900v3#page=2)。圖說未收錄。*\n\n最左邊的 MetaFormer 區塊有兩條殘差連接，token mixer 一條、MLP 一條，Transformer 就是這種結構。ConvNeXt 的區塊只有一條殘差，空間方向的混合、正規化和 MLP 串在同一條路上。作者把這種單殘差的框架叫做 MetaNeXt，ConvNeXt 和這篇的 InceptionNeXt 都是它的一種實作，差別只在 token mixer 用什麼。\n\n### 慢在哪裡\n\n作者在 ConvNeXt-T 上做了一組預備實驗，只動 depthwise 卷積，其他不變：\n\n| depthwise 卷積核 | 做卷積的通道比例 | 訓練（張／秒） | 推論（張／秒） | top-1 |\n| --- | ---: | ---: | ---: | ---: |\n| 7×7 | 1 | 575 | 2413 | 82.1% |\n| 5×5 | 1 | 675 | 2704 | 82.0% |\n| 3×3 | 1 | 798 | 2802 | 81.5% |\n| 3×3 | 1/4 | 871 | 2808 | 81.3% |\n| 3×3 | 1/8 | 901 | 2833 | 80.8% |\n| 3×3 | 1/16 | 916 | 2846 | 80.1% |\n\n*表：摘自原論文 Table 1 的部分列。A100，批次 128，TF32。*\n\n卷積核從 7×7 縮到 3×3，訓練快了 39%，準確率掉了 0.6 個百分點。接著作者學 ShuffleNet V2，只讓一部分通道做 depthwise 卷積，其餘原封不動。比例降到 1/4，訓練又快了一些，準確率只再掉 0.2。\n\n兩個發現由此而來。大卷積核的 depthwise 卷積是速度瓶頸；一層 depthwise 卷積只處理一部分通道就夠了。問題是，卷積核縮小以後，感受野也跟著縮小了，那 0.6 個百分點要從哪裡拿回來？\n\n## 解決問題\n\n### Inception depthwise 卷積\n\n作者把通道切成四份，每份做不同的事：\n\n- 一份做 3×3 的 depthwise 卷積\n- 一份做 1×11 的水平長條卷積\n- 一份做 11×1 的垂直長條卷積\n- 剩下的原封不動\n\n三條卷積分支各分到 $C/8$ 個通道，剩下的 $5C/8$ 直接通過，最後把四份接回去。把 $k\\times k$ 拆成 $1\\times k$ 和 $k\\times 1$，是 Inception-v3 用過的老招，名字也就這樣來了。\n\n\n> [!TIP]\n> \n> Inception-v3 當年用這招把大卷積核拆掉，可以看這篇：\n> \n> - **[[15.12] Inception-v3：大卷積核退場](/blog/papers/classic-cnns/inception-v3)**\n\n\n參數量從 depthwise 卷積的 $k^2C$ 變成 $(2k+9)C/8$，$2k$ 來自兩條長條，9 來自 3×3。和卷積核大小的關係從平方變成線性。我們照論文附的 PyTorch 程式寫一遍，數數參數：\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass InceptionDWConv2d(nn.Module):\n    def __init__(self, c, square=3, band=11, ratio=1/8):\n        super().__init__()\n        g = int(c * ratio)                                    # 每條卷積分支的通道數\n        self.hw = nn.Conv2d(g, g, square, padding=square // 2, groups=g)\n        self.w = nn.Conv2d(g, g, (1, band), padding=(0, band // 2), groups=g)\n        self.h = nn.Conv2d(g, g, (band, 1), padding=(band // 2, 0), groups=g)\n        self.split = (g, g, g, c - 3 * g)                     # 3×3、1×11、11×1、原封不動\n\n    def forward(self, x):\n        x_hw, x_w, x_h, x_id = torch.split(x, self.split, dim=1)\n        return torch.cat((self.hw(x_hw), self.w(x_w), self.h(x_h), x_id), dim=1)\n\nc = 96                                                        # InceptionNeXt-T 第一階段的通道數\nparams = lambda m: sum(p.numel() for p in m.parameters())\nprint(\"7×7 depthwise：\", params(nn.Conv2d(c, c, 7, padding=3, groups=c)))\nprint(\"3×3 depthwise：\", params(nn.Conv2d(c, c, 3, padding=1, groups=c)))\nprint(\"Inception depthwise：\", params(InceptionDWConv2d(c)))\n```\n\n```text\n7×7 depthwise： 4800\n3×3 depthwise： 960\nInception depthwise： 408\n```\n\nInception depthwise 卷積的參數比 3×3 depthwise 還少，因為八分之五的通道根本沒做卷積。\n\n### 十字形的感受野\n\n看到這裡，你可能會覺得有點虧。7×7 卷積核的每個通道都看得到一整塊 7×7，現在每個通道只看得到一個 3×3 小方塊，或是一條橫線、一條直線。\n\n[GitHub 上有人問過](https://github.com/sail-sg/inceptionnext/issues/3)這個「拆解」和 7×7 是不是數學上等價，作者回答不是，這裡的 decompose 比較接近「替換」。那替換之後到底看得到多大的範圍？\n\n關鍵在 MLP。depthwise 卷積之後接的 1×1 卷積會把通道混在一起，上一個區塊做過水平長條的特徵，到了下一個區塊可能被分去做垂直長條。我們把區塊簡化成「空間卷積接 1×1 卷積」，用梯度回推中心那個輸出看得到哪些輸入位置：\n\n```python\ndef seen(blocks, make_mixer, size=33):\n    torch.manual_seed(0)\n    layers = []\n    for _ in range(blocks):                                   # 每個區塊：空間卷積，再接 1×1 卷積混合通道\n        layers += [make_mixer(), nn.Conv2d(c, c, 1)]\n    net = nn.Sequential(*layers)\n    x = torch.randn(1, c, size, size, requires_grad=True)\n    net(x)[0, :, size // 2, size // 2].sum().backward()       # 中心那個輸出看得到哪些輸入位置\n    grid = x.grad[0].abs().sum(0) > 0\n    rows, cols = grid.nonzero().T\n    return f\"看得到 {int(grid.sum())} 個位置，範圍 {int(rows.max() - rows.min() + 1)}×{int(cols.max() - cols.min() + 1)}\"\n\nfor blocks in (1, 2):\n    print(f\"疊 {blocks} 個區塊\")\n    print(\"  7×7 depthwise：\", seen(blocks, lambda: nn.Conv2d(c, c, 7, padding=3, groups=c)))\n    print(\"  Inception depthwise：\", seen(blocks, lambda: InceptionDWConv2d(c)))\n```\n\n```text\n疊 1 個區塊\n  7×7 depthwise： 看得到 49 個位置，範圍 7×7\n  Inception depthwise： 看得到 25 個位置，範圍 11×11\n疊 2 個區塊\n  7×7 depthwise： 看得到 169 個位置，範圍 13×13\n  Inception depthwise： 看得到 149 個位置，範圍 21×21\n```\n\n只疊一個區塊時，Inception depthwise 看到的是一個十字，3×3 的中心加上兩條 11 格長的線，共 25 個位置。疊到第二個區塊，水平線接上垂直線，就補成一整塊 11×11，再加上往外伸的四條手臂，總共 149 個位置。7×7 疊兩層是 13×13 共 169 個位置，數量差不多，涵蓋的範圍反而比較小。\n\n### 組成網路\n\n各階段的區塊數和通道數沿用 ConvNeXt，四個階段的結構也一樣。不同的地方有兩處。正規化用 BatchNorm，因為這篇講求速度，推論時 BN 可以併進卷積；最後一個階段的 MLP 擴張倍數從 4 降到 3，省下來的參數挪給分類頭。最小的 Atto 版本把每條分支的比例提高到 1/4，長條長度改成 9。\n\n## 討論\n\n### ImageNet-1K\n\n訓練設定沿用 DeiT 和 ConvNeXt 的配方，訓練 300 個 epoch。吞吐量在 A100 上用 TF32 量：\n\n| 模型 | 參數量 | MACs | 訓練（張／秒） | 推論（張／秒） | top-1 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| RSB-ResNet-50 | 26M | 4.1G | 969 | 3149 | 79.8% |\n| ConvNeXt-T | 29M | 4.5G | 575 | 2413 | 82.1% |\n| InceptionNeXt-T | 28M | 4.2G | **901** | **2900** | **82.3%** |\n| ConvNeXt-S | 50M | 8.7G | 361 | 1535 | 83.1% |\n| InceptionNeXt-S | 49M | 8.4G | **521** | **1750** | **83.5%** |\n| ConvNeXt-B | 89M | 15.4G | 267 | 1122 | 83.8% |\n| InceptionNeXt-B | 87M | 14.9G | **375** | **1244** | **84.0%** |\n\n*表：摘自原論文 Table 4 的部分列。RSB-ResNet-50 是用「ResNet strikes back」新配方重新訓練的 ResNet-50。*\n\nInceptionNeXt-T 比 ConvNeXt-T 高 0.2 個百分點，訓練快 57%，推論快 20%。和 ResNet-50 比，訓練速度到了九成三，推論是九成二，說「ResNet-50 的速度」稍微樂觀了一點，不過已經很接近。\n\n加速幅度隨模型變大而縮小，T 的訓練快 57%，S 剩 44%，B 剩 40%。原因是 depthwise 卷積的計算量和通道數成正比，MLP 則和通道數的平方成正比。模型越寬，時間越花在 MLP 上，只改 token mixer 能省的比例就越小。\n\n最小的 Atto 版本反過來，訓練快了 219%，準確率卻比 ConvNeXt-A 低 0.4 個百分點。\n\n速度數字也和硬體有關。同一張表在 2080Ti 上用 FP32 量，T 的推論快了 39%，比 A100 上的 20% 多。後來 RDNet 的論文自己量過一次推論延遲，每批 128 張，InceptionNeXt-T 要 132 ms，ConvNeXt-T 要 150 ms，換算下來快 14%。\n\n### MetaNeXt 裝不進注意力\n\n既然 MetaNeXt 是 MetaFormer 的簡化版，作者也試了把自注意力放進單殘差的區塊，做成 MetaNeXt-Attn。結果訓練不起來，ImageNet 準確率只有 3.9%，同樣大小的 DeiT-S 是 79.8%。\n\n兩條殘差合成一條，速度是快了，可是 token mixer 不能太複雜。這個框架能裝卷積，裝不進注意力。\n\n### 語意分割\n\nADE20K 上接 UperNet，訓練 160K 次：\n\n| 尺寸 | Swin | ConvNeXt | InceptionNeXt |\n| --- | ---: | ---: | ---: |\n| T | 45.8 | 46.7 | **47.9** |\n| S | 49.5 | 49.6 | **50.0** |\n| B | 49.7 | 49.9 | **50.6** |\n\n*表：摘自原論文 Table 6，數字是 mIoU。*\n\nT 的差距是 1.2，比分類的 0.2 大不少。不過要注意測試方式。[有人在 GitHub 上回報](https://github.com/sail-sg/inceptionnext/issues/10)，把 InceptionNeXt 接進自己的分割模型，結果和 ConvNeXt 差不多；作者回覆，表中沿用 ConvNeXt 的做法，報的是多尺度測試的 mIoU，單尺度測試時 InceptionNeXt 和 ConvNeXt 相近。分割的訓練程式碼也沒有釋出，issue 裡有好幾位使用者在問。\n\n### 消融\n\n以 InceptionNeXt-T 為基準，每次改一個地方：\n\n| 改動 | 訓練（張／秒） | 推論（張／秒） | top-1 |\n| --- | ---: | ---: | ---: |\n| 基準 | 901 | 2900 | 82.3% |\n| 拿掉水平長條 | 947 | 3093 | 81.9% |\n| 拿掉垂直長條 | 954 | 3173 | 81.9% |\n| 拿掉 3×3 方塊 | 940 | 3004 | 82.0% |\n| 兩條長條改成串接 | 903 | 2971 | 82.1% |\n| 長條從 11 改成 13 | 896 | 2895 | 82.0% |\n| 分支比例 1/8 改成 1/4 | 834 | 2499 | 82.2% |\n| 分支比例 1/8 改成 1/16 | 936 | 3097 | 81.8% |\n| BatchNorm 改成 LayerNorm | 721 | 2646 | 82.4% |\n\n*表：摘自原論文 Table 8 的部分列。*\n\n兩條長條最重要，拿掉任何一條都掉 0.4，感受野是靠它們撐起來的。3×3 方塊拿掉只掉 0.3，作者建議如果更在意速度，可以用沒有方塊的版本。分支比例加大到 1/4 沒有幫助，只是變慢；降到 1/16 就掉了 0.5。\n\nLayerNorm 多了 0.1 個百分點，訓練速度卻少了兩成，作者選了 BatchNorm。\n\n## 結論\n\nInceptionNeXt 把 7×7 depthwise 卷積換成一個小方塊、兩條長條和一大群不做事的通道。ConvNeXt-T 的準確率保住了，訓練速度拉回 ResNet-50 附近。\n\n它只用到 PyTorch 內建的 depthwise 卷積，不需要自訂運算子，timm 裡也有 `inception_next_tiny` 等預訓練權重。想加速自己的 ConvNeXt 類模型，把 token mixer 換掉就能試。\n\n至於分割表上那 1.2 個百分點，先問一句是單尺度還是多尺度，再決定要不要換骨幹。\n","imageSizes":{"/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png":{"width":924,"height":666},"/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png":{"width":1446,"height":786}}}