{"notes":[{"id":"classic-cnns/inceptionnext","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InceptionNeXt","subtitle":"拆成十字","author":"Z. Yuan","paperDate":"2023-03-29","publishedOn":"2025-10-30","world":"external/inceptionnext","paperTitle":"InceptionNeXt: When Inception Meets ConvNeXt","paperUrl":"https://arxiv.org/abs/2303.16900","bibliography":{"title":"InceptionNeXt: When Inception Meets ConvNeXt","authors":["Weihao Yu","Pan Zhou","Shuicheng Yan","Xinchao Wang"],"url":"https://arxiv.org/abs/2303.16900","pdfUrl":"https://arxiv.org/pdf/2303.16900v3","codeUrl":"https://github.com/sail-sg/inceptionnext","preprint":{"id":"2303.16900","submittedOn":"2023-03-29","url":"https://arxiv.org/abs/2303.16900"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Yu_InceptionNeXt_When_Inception_Meets_ConvNeXt_CVPR_2024_paper.html"},"citation":"@misc{classiccnnsinceptionnext2023,\n  title = {{InceptionNeXt: When Inception Meets ConvNeXt}},\n  author = {Weihao Yu and Pan Zhou and Shuicheng Yan and Xinchao Wang},\n  year = {2023},\n  eprint = {2303.16900},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2303.16900}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inceptionnext.82116f5daa6e.bib"},"thumbnail":{"src":"/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png","width":1446,"height":786},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inceptionnext.3a6b71210e12.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"metanext","text":"MetaNeXt","depth":3},{"id":"慢在哪里","text":"慢在哪里","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"inception-depthwise-卷积","text":"Inception depthwise 卷积","depth":3},{"id":"十字形的感受野","text":"十字形的感受野","depth":3},{"id":"组成网络","text":"组成网络","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet-1k","text":"ImageNet-1K","depth":3},{"id":"metanext-装不进注意力","text":"MetaNeXt 装不进注意力","depth":3},{"id":"语义分割","text":"语义分割","depth":3},{"id":"消融","text":"消融","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"vision-transformers/poolformer","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"让卷积再次伟大","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/convnext.db5ced84300e.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"重新设计网络架构","text":"重新设计网络架构","depth":3},{"id":"引入现代训练技术","text":"引入现代训练技术","depth":3},{"id":"宏观设计","text":"宏观设计","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"大核卷积","text":"大核卷积","depth":3},{"id":"微观设计","text":"微观设计","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"总结模型配置","text":"总结模型配置","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"用卷积混合图像分块"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"用卷积混合图像分块"},{"id":"reparameterization/replknet","relation":"comparison","topic":"大卷积核与简化骨架"}]},{"id":"vision-transformers/poolformer","category":"vision-transformers","legacyPath":"vision-transformers/2111-poolformer","legacySlug":"2111-poolformer","legacyPaths":[],"title":"PoolFormer","subtitle":"你需要的是框架！","author":"Z. Yuan","paperDate":"2021-11-22","publishedOn":"2024-07-10","paperTitle":"MetaFormer Is Actually What You Need for Vision","paperUrl":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","bibliography":{"title":"MetaFormer Is Actually What You Need for Vision","authors":["Weihao Yu","Mi Luo","Pan Zhou","Chenyang Si","Yichen Zhou","Xinchao Wang","Jiashi Feng","Shuicheng Yan"],"url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2022/papers/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.pdf","codeUrl":"https://github.com/sail-sg/poolformer","preprint":{"id":"2111.11418","submittedOn":"2021-11-22","url":"https://arxiv.org/abs/2111.11418"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html"},"citation":"@misc{visiontransformerspoolformer2021,\n  title = {{MetaFormer Is Actually What You Need for Vision}},\n  author = {Weihao Yu and Mi Luo and Pan Zhou and Chenyang Si and Yichen Zhou and Xinchao Wang and Jiashi Feng and Shuicheng Yan},\n  year = {2021},\n  eprint = {2111.11418},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.11418}\n}\n","citationPath":"/content/papers/citations/vision-transformers/poolformer.f2f96629d283.bib"},"thumbnail":{"src":"/content/papers/assets/3cb141e9bb1247410c8214276b21b28ed1d536060d470406f1bfb0c88fe68df2.jpg","width":1498,"height":548},"locale":"zh-hans","path":"/content/papers/zh-hans/vision-transformers/poolformer.1460bbffaf4a.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"input-embedding","text":"Input Embedding","depth":3},{"id":"token-mixer","text":"Token-Mixer","depth":3},{"id":"channel-mlp","text":"Channel MLP","depth":3},{"id":"或许-token-mixer-不是重点","text":"或许 Token-Mixer 不是重点？","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型架构","text":"模型架构","depth":3},{"id":"模型配置","text":"模型配置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet-实验","text":"ImageNet 实验","depth":3},{"id":"消融实验","text":"消融实验","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/mlp-mixer","relation":"background","topic":"把骨架与 Token 混合方式分开看"},{"id":"vision-transformers/caformer","relation":"next","topic":"把骨架与 Token 混合方式分开看"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"大卷积核退场","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2023-11-15","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html"},"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inception-v3.2087fa7b4a8e.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"把-55-拆成两层-33","text":"把 5×5 拆成两层 3×3","depth":3},{"id":"再拆成-1n-与-n1","text":"再拆成 1×n 与 n×1","depth":3},{"id":"辅助分类器的真正作用","text":"辅助分类器的真正作用","depth":3},{"id":"缩小特征图又不想卡住信息","text":"缩小特征图，又不想卡住信息","depth":3},{"id":"把零件组装起来","text":"把零件组装起来","depth":3},{"id":"让模型别太有把握","text":"让模型别太有把握","depth":3},{"id":"训练设置","text":"训练设置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"一项一项加上去","text":"一项一项加上去","depth":3},{"id":"分辨率低一点差多少","text":"分辨率低一点，差多少？","depth":3},{"id":"和其他模型比较","text":"和其他模型比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]}],"text":"ConvNeXt-T 的计算量是 4.5 GMACs，ResNet-50 是 4.1 GMACs，两者差不多。放到 A100 上训练，ResNet-50 每秒可以处理 969 张图，ConvNeXt-T 只有 575 张，大约六成。\n\n计算量差不多，速度却差这么多，问题出在 7×7 的 depthwise 卷积。它每个通道各算各的，乘加次数很少，要搬的数据却不少，GPU 大部分时间都在等内存。ConvNeXt 把卷积核从 3×3 放大到 7×7，换来了准确率，也换来这个速度问题。\n\n这篇来自新加坡国立大学、新加坡管理大学、Sea AI Lab 和 Skywork AI。第一作者 Weihao Yu 也是 MetaFormer 的作者，当时他把 Transformer 拆成「token mixer 加 MLP」的框架；这次同样的拆法用在 ConvNeXt 上。\n\n\n> [!TIP]\n> \n> ConvNeXt 和 MetaFormer 可以先看这两篇：\n> \n> - **[[22.01] ConvNeXt：让卷积再次伟大](/blog/papers/classic-cnns/convnext)**\n> - **[[21.11] PoolFormer：你需要的是框架！](/blog/papers/vision-transformers/poolformer)**\n\n\n[![原论文 Figure 1：横轴是 A100 上的训练吞吐量（张／秒），纵轴是 ImageNet top-1 准确率。ConvNeXt-T/k7 在左上方约 82.1%、575 张／秒；换成 5×5 和 3×3 卷积核后往右移，准确率逐步下降；ResNet-50 在右下角，速度约 970 张／秒但准确率只有 78% 多；InceptionNeXt-T 在右上方，红色箭头标示比 ConvNeXt-T/k7 快 1.6 倍。](/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png)](/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png)\n\n*原论文 Figure 1（Yu 等人），取自 [**PDF 第 1 页**](https://arxiv.org/pdf/2303.16900v3#page=1)。原图注未收录。*\n\n图的右上角是这篇要的位置，准确率不输 ConvNeXt-T，速度接近 ResNet-50。\n\n## 定义问题\n\n### MetaNeXt\n\n作者先把 ConvNeXt 的区块抽象化。\n\n[![原论文 Figure 2：四个区块由左到右排列。MetaFormer 区块有两条残差连接，第一条包住 Norm 和虚线框的 Token Mixer，第二条包住 Norm 和 MLP；MetaNeXt 区块只有一条残差连接，Token Mixer、Norm、MLP 串在同一条路上；ConvNeXt 区块把 Token Mixer 换成 7×7 的 DWConv；InceptionNeXt 区块的 Token Mixer 是一个虚线框，输入先 Split 成四份，分别经过 3×3、1×11、11×1 的 DWConv 和 Identity，再 Concat 起来。](/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png)](/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png)\n\n*原论文 Figure 2（Yu 等人），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/2303.16900v3#page=2)。原图注未收录。*\n\n最左边的 MetaFormer 区块有两条残差连接，token mixer 一条、MLP 一条，Transformer 就是这种结构。ConvNeXt 的区块只有一条残差，空间方向的混合、归一化和 MLP 串在同一条路上。作者把这种单残差的框架叫做 MetaNeXt，ConvNeXt 和这篇的 InceptionNeXt 都是它的一种实现，差别只在 token mixer 用什么。\n\n### 慢在哪里\n\n作者在 ConvNeXt-T 上做了一组预备实验，只动 depthwise 卷积，其他不变：\n\n| depthwise 卷积核 | 做卷积的通道比例 | 训练（张／秒） | 推理（张／秒） | top-1 |\n| --- | ---: | ---: | ---: | ---: |\n| 7×7 | 1 | 575 | 2413 | 82.1% |\n| 5×5 | 1 | 675 | 2704 | 82.0% |\n| 3×3 | 1 | 798 | 2802 | 81.5% |\n| 3×3 | 1/4 | 871 | 2808 | 81.3% |\n| 3×3 | 1/8 | 901 | 2833 | 80.8% |\n| 3×3 | 1/16 | 916 | 2846 | 80.1% |\n\n*表：摘自原论文 Table 1 的部分行。A100，批量 128，TF32。*\n\n卷积核从 7×7 缩到 3×3，训练快了 39%，准确率掉了 0.6 个百分点。接着作者学 ShuffleNet V2，只让一部分通道做 depthwise 卷积，其余原封不动。比例降到 1/4，训练又快了一些，准确率只再掉 0.2。\n\n两个发现由此而来。大卷积核的 depthwise 卷积是速度瓶颈；一层 depthwise 卷积只处理一部分通道就够了。问题是，卷积核缩小以后，感受野也跟着缩小了，那 0.6 个百分点要从哪里拿回来？\n\n## 解决问题\n\n### Inception depthwise 卷积\n\n作者把通道切成四份，每份做不同的事：\n\n- 一份做 3×3 的 depthwise 卷积\n- 一份做 1×11 的水平长条卷积\n- 一份做 11×1 的垂直长条卷积\n- 剩下的原封不动\n\n三条卷积分支各分到 $C/8$ 个通道，剩下的 $5C/8$ 直接通过，最后把四份接回去。把 $k\\times k$ 拆成 $1\\times k$ 和 $k\\times 1$，是 Inception-v3 用过的老招，名字也就这样来了。\n\n\n> [!TIP]\n> \n> Inception-v3 当年用这招把大卷积核拆掉，可以看这篇：\n> \n> - **[[15.12] Inception-v3：大卷积核退场](/blog/papers/classic-cnns/inception-v3)**\n\n\n参数量从 depthwise 卷积的 $k^2C$ 变成 $(2k+9)C/8$，$2k$ 来自两条长条，9 来自 3×3。和卷积核大小的关系从平方变成线性。我们照论文附的 PyTorch 程序写一遍，数数参数：\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass InceptionDWConv2d(nn.Module):\n    def __init__(self, c, square=3, band=11, ratio=1/8):\n        super().__init__()\n        g = int(c * ratio)                                    # 每条卷积分支的通道数\n        self.hw = nn.Conv2d(g, g, square, padding=square // 2, groups=g)\n        self.w = nn.Conv2d(g, g, (1, band), padding=(0, band // 2), groups=g)\n        self.h = nn.Conv2d(g, g, (band, 1), padding=(band // 2, 0), groups=g)\n        self.split = (g, g, g, c - 3 * g)                     # 3×3、1×11、11×1、原封不动\n\n    def forward(self, x):\n        x_hw, x_w, x_h, x_id = torch.split(x, self.split, dim=1)\n        return torch.cat((self.hw(x_hw), self.w(x_w), self.h(x_h), x_id), dim=1)\n\nc = 96                                                        # InceptionNeXt-T 第一阶段的通道数\nparams = lambda m: sum(p.numel() for p in m.parameters())\nprint(\"7×7 depthwise：\", params(nn.Conv2d(c, c, 7, padding=3, groups=c)))\nprint(\"3×3 depthwise：\", params(nn.Conv2d(c, c, 3, padding=1, groups=c)))\nprint(\"Inception depthwise：\", params(InceptionDWConv2d(c)))\n```\n\n```text\n7×7 depthwise： 4800\n3×3 depthwise： 960\nInception depthwise： 408\n```\n\nInception depthwise 卷积的参数比 3×3 depthwise 还少，因为八分之五的通道根本没做卷积。\n\n### 十字形的感受野\n\n看到这里，你可能会觉得有点亏。7×7 卷积核的每个通道都看得到一整块 7×7，现在每个通道只看得到一个 3×3 小方块，或是一条横线、一条直线。\n\n[GitHub 上有人问过](https://github.com/sail-sg/inceptionnext/issues/3)这个「拆解」和 7×7 是不是数学上等价，作者回答不是，这里的 decompose 比较接近「替换」。那替换之后到底看得到多大的范围？\n\n关键在 MLP。depthwise 卷积之后接的 1×1 卷积会把通道混在一起，上一个区块做过水平长条的特征，到了下一个区块可能被分去做垂直长条。我们把区块简化成「空间卷积接 1×1 卷积」，用梯度回推中心那个输出看得到哪些输入位置：\n\n```python\ndef seen(blocks, make_mixer, size=33):\n    torch.manual_seed(0)\n    layers = []\n    for _ in range(blocks):                                   # 每个区块：空间卷积，再接 1×1 卷积混合通道\n        layers += [make_mixer(), nn.Conv2d(c, c, 1)]\n    net = nn.Sequential(*layers)\n    x = torch.randn(1, c, size, size, requires_grad=True)\n    net(x)[0, :, size // 2, size // 2].sum().backward()       # 中心那个输出看得到哪些输入位置\n    grid = x.grad[0].abs().sum(0) > 0\n    rows, cols = grid.nonzero().T\n    return f\"看得到 {int(grid.sum())} 个位置，范围 {int(rows.max() - rows.min() + 1)}×{int(cols.max() - cols.min() + 1)}\"\n\nfor blocks in (1, 2):\n    print(f\"叠 {blocks} 个区块\")\n    print(\"  7×7 depthwise：\", seen(blocks, lambda: nn.Conv2d(c, c, 7, padding=3, groups=c)))\n    print(\"  Inception depthwise：\", seen(blocks, lambda: InceptionDWConv2d(c)))\n```\n\n```text\n叠 1 个区块\n  7×7 depthwise： 看得到 49 个位置，范围 7×7\n  Inception depthwise： 看得到 25 个位置，范围 11×11\n叠 2 个区块\n  7×7 depthwise： 看得到 169 个位置，范围 13×13\n  Inception depthwise： 看得到 149 个位置，范围 21×21\n```\n\n只叠一个区块时，Inception depthwise 看到的是一个十字，3×3 的中心加上两条 11 格长的线，共 25 个位置。叠到第二个区块，水平线接上垂直线，就补成一整块 11×11，再加上往外伸的四条手臂，总共 149 个位置。7×7 叠两层是 13×13 共 169 个位置，数量差不多，涵盖的范围反而比较小。\n\n### 组成网络\n\n各阶段的区块数和通道数沿用 ConvNeXt，四个阶段的结构也一样。不同的地方有两处。归一化用 BatchNorm，因为这篇讲求速度，推理时 BN 可以并入卷积；最后一个阶段的 MLP 扩张倍数从 4 降到 3，省下来的参数挪给分类头。最小的 Atto 版本把每条分支的比例提高到 1/4，长条长度改成 9。\n\n## 讨论\n\n### ImageNet-1K\n\n训练设置沿用 DeiT 和 ConvNeXt 的配方，训练 300 个 epoch。吞吐量在 A100 上用 TF32 量：\n\n| 模型 | 参数量 | MACs | 训练（张／秒） | 推理（张／秒） | top-1 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| RSB-ResNet-50 | 26M | 4.1G | 969 | 3149 | 79.8% |\n| ConvNeXt-T | 29M | 4.5G | 575 | 2413 | 82.1% |\n| InceptionNeXt-T | 28M | 4.2G | **901** | **2900** | **82.3%** |\n| ConvNeXt-S | 50M | 8.7G | 361 | 1535 | 83.1% |\n| InceptionNeXt-S | 49M | 8.4G | **521** | **1750** | **83.5%** |\n| ConvNeXt-B | 89M | 15.4G | 267 | 1122 | 83.8% |\n| InceptionNeXt-B | 87M | 14.9G | **375** | **1244** | **84.0%** |\n\n*表：摘自原论文 Table 4 的部分行。RSB-ResNet-50 是用「ResNet strikes back」新配方重新训练的 ResNet-50。*\n\nInceptionNeXt-T 比 ConvNeXt-T 高 0.2 个百分点，训练快 57%，推理快 20%。和 ResNet-50 比，训练速度到了九成三，推理是九成二，说「ResNet-50 的速度」稍微乐观了一点，不过已经很接近。\n\n加速幅度随模型变大而缩小，T 的训练快 57%，S 剩 44%，B 剩 40%。原因是 depthwise 卷积的计算量和通道数成正比，MLP 则和通道数的平方成正比。模型越宽，时间越花在 MLP 上，只改 token mixer 能省的比例就越小。\n\n最小的 Atto 版本反过来，训练快了 219%，准确率却比 ConvNeXt-A 低 0.4 个百分点。\n\n速度数字也和硬件有关。同一张表在 2080Ti 上用 FP32 量，T 的推理快了 39%，比 A100 上的 20% 多。后来 RDNet 的论文自己量过一次推理延迟，每批 128 张，InceptionNeXt-T 要 132 ms，ConvNeXt-T 要 150 ms，换算下来快 14%。\n\n### MetaNeXt 装不进注意力\n\n既然 MetaNeXt 是 MetaFormer 的简化版，作者也试了把自注意力放进单残差的区块，做成 MetaNeXt-Attn。结果训练不起来，ImageNet 准确率只有 3.9%，同样大小的 DeiT-S 是 79.8%。\n\n两条残差合成一条，速度是快了，可是 token mixer 不能太复杂。这个框架能装卷积，装不进注意力。\n\n### 语义分割\n\nADE20K 上接 UperNet，训练 160K 次：\n\n| 尺寸 | Swin | ConvNeXt | InceptionNeXt |\n| --- | ---: | ---: | ---: |\n| T | 45.8 | 46.7 | **47.9** |\n| S | 49.5 | 49.6 | **50.0** |\n| B | 49.7 | 49.9 | **50.6** |\n\n*表：摘自原论文 Table 6，数字是 mIoU。*\n\nT 的差距是 1.2，比分类的 0.2 大不少。不过要注意测试方式。[有人在 GitHub 上反馈](https://github.com/sail-sg/inceptionnext/issues/10)，把 InceptionNeXt 接进自己的分割模型，结果和 ConvNeXt 差不多；作者回复，表中沿用 ConvNeXt 的做法，报的是多尺度测试的 mIoU，单尺度测试时 InceptionNeXt 和 ConvNeXt 相近。分割的训练代码也没有发布，issue 里有好几位用户在问。\n\n### 消融\n\n以 InceptionNeXt-T 为基准，每次改一个地方：\n\n| 改动 | 训练（张／秒） | 推理（张／秒） | top-1 |\n| --- | ---: | ---: | ---: |\n| 基准 | 901 | 2900 | 82.3% |\n| 拿掉水平长条 | 947 | 3093 | 81.9% |\n| 拿掉垂直长条 | 954 | 3173 | 81.9% |\n| 拿掉 3×3 方块 | 940 | 3004 | 82.0% |\n| 两条长条改成串联 | 903 | 2971 | 82.1% |\n| 长条从 11 改成 13 | 896 | 2895 | 82.0% |\n| 分支比例 1/8 改成 1/4 | 834 | 2499 | 82.2% |\n| 分支比例 1/8 改成 1/16 | 936 | 3097 | 81.8% |\n| BatchNorm 改成 LayerNorm | 721 | 2646 | 82.4% |\n\n*表：摘自原论文 Table 8 的部分行。*\n\n两条长条最重要，拿掉任何一条都掉 0.4，感受野是靠它们撑起来的。3×3 方块拿掉只掉 0.3，作者建议如果更在意速度，可以用没有方块的版本。分支比例加大到 1/4 没有帮助，只是变慢；降到 1/16 就掉了 0.5。\n\nLayerNorm 多了 0.1 个百分点，训练速度却少了两成，作者选了 BatchNorm。\n\n## 结论\n\nInceptionNeXt 把 7×7 depthwise 卷积换成一个小方块、两条长条和一大群不做事的通道。ConvNeXt-T 的准确率保住了，训练速度拉回 ResNet-50 附近。\n\n它只用到 PyTorch 内置的 depthwise 卷积，不需要自定义算子，timm 里也有 `inception_next_tiny` 等预训练权重。想加速自己的 ConvNeXt 类模型，把 token mixer 换掉就能试。\n\n至于分割表上那 1.2 个百分点，先问一句是单尺度还是多尺度，再决定要不要换骨干。\n","imageSizes":{"/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png":{"width":924,"height":666},"/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png":{"width":1446,"height":786}}}