{"notes":[{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"卷积核 Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/slak.d84cfaf376c9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"31-之后","text":"31 之后","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"第一步拆成两根长条","text":"第一步：拆成两根长条","depth":3},{"id":"第二步稀疏一点宽一点","text":"第二步：稀疏一点，宽一点","depth":3},{"id":"slak-的样子","text":"SLaK 的样子","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任务","text":"下游任务","depth":3},{"id":"实际跑起来","text":"实际跑起来","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"让卷积再次伟大","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/convnext.db5ced84300e.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"重新设计网络架构","text":"重新设计网络架构","depth":3},{"id":"引入现代训练技术","text":"引入现代训练技术","depth":3},{"id":"宏观设计","text":"宏观设计","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"大核卷积","text":"大核卷积","depth":3},{"id":"微观设计","text":"微观设计","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"总结模型配置","text":"总结模型配置","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"用卷积混合图像分块"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"用卷积混合图像分块"},{"id":"reparameterization/replknet","relation":"comparison","topic":"大卷积核与简化骨架"}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"巨无霸卷积核","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"zh-hans","path":"/content/papers/zh-hans/reparameterization/replknet.63f1e846afbb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"大核卷积使用说明书","text":"大核卷积使用说明书","depth":2},{"id":"深度大核卷积仍然高效","text":"深度大核卷积仍然高效","depth":3},{"id":"大核卷积十分依赖残差连接","text":"大核卷积十分依赖残差连接","depth":3},{"id":"大核卷积需要重参数化","text":"大核卷积需要重参数化","depth":3},{"id":"大核卷积明显提升下游任务性能","text":"大核卷积明显提升下游任务性能","depth":3},{"id":"大核卷积对于小特征图的效果依然很好","text":"大核卷积对于小特征图的效果依然很好","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"网络架构","text":"网络架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"大核卷积评估","text":"大核卷积评估","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"大核-cnn-比深度小核模型有更大的-erf","text":"大核 CNN 比深度小核模型有更大的 ERF","depth":3},{"id":"形状偏差","text":"形状偏差","depth":3},{"id":"限制","text":"限制","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"大卷积核与简化骨架"}]},{"id":"classic-cnns/resnext","category":"classic-cnns","legacyPath":"classic-cnns/1611-resnext","legacySlug":"1611-resnext","legacyPaths":[],"title":"ResNeXt","subtitle":"探访下一个维度","author":"Z. Yuan","paperDate":"2016-11-16","publishedOn":"2024-07-12","paperTitle":"Aggregated Residual Transformations for Deep Neural Networks","paperUrl":"https://arxiv.org/abs/1611.05431","bibliography":{"title":"Aggregated Residual Transformations for Deep Neural Networks","authors":["Saining Xie","Ross Girshick","Piotr Dollár","Zhuowen Tu","Kaiming He"],"url":"https://arxiv.org/abs/1611.05431","pdfUrl":"https://arxiv.org/pdf/1611.05431","codeUrl":"https://github.com/facebookresearch/ResNeXt","preprint":{"id":"1611.05431","submittedOn":"2016-11-16","url":"https://arxiv.org/abs/1611.05431"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1611.05431"},"citation":"@misc{classiccnnsresnext2016,\n  title = {{Aggregated Residual Transformations for Deep Neural Networks}},\n  author = {Saining Xie and Ross Girshick and Piotr Dollár and Zhuowen Tu and Kaiming He},\n  year = {2016},\n  eprint = {1611.05431},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1611.05431}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnext.4945bb6b3dae.bib"},"thumbnail":{"src":"/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg","width":1224,"height":576},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnext.b391d1bf5dbf.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"下游任务泛化性","text":"下游任务泛化性","depth":3},{"id":"不利于模型设计","text":"不利于模型设计","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"网络架构","text":"网络架构","depth":3},{"id":"模型配置","text":"模型配置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet-1k-训练效果","text":"ImageNet-1K 训练效果","depth":3},{"id":"增加模型复杂度","text":"增加模型复杂度","depth":3},{"id":"imagenet-1k-模型表现","text":"ImageNet-1K 模型表现","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分组卷积与特征复用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分组卷积与特征复用"}]}],"text":"2022 年上半年，卷积网络接连出了两篇翻身之作。ConvNeXt 照着 Swin Transformer 的设计一步步改 ResNet，卷积核从 3×3 放大到 7×7；两个月后 RepLKNet 更大胆，直接用 31×31，再靠重参数化的小卷积核帮忙训练。\n\n\n> [!TIP]\n> \n> 这两篇可以先看我们的笔记：\n> \n> - **[[22.01] ConvNeXt：让卷积再次伟大](/blog/papers/classic-cnns/convnext)**\n> - **[[22.03] RepLKNet：巨无霸卷积核](/blog/papers/reparameterization/replknet)**\n\n\nRepLKNet 停在 31×31。这篇论文想知道卷积核还能不能更大，放到 51×51、61×61 时准确率会不会继续往上走。作者团队来自 UT Austin、埃因霍温理工大学等学校。Decebal Mocanu 和第一作者 Shiwei Liu 长年研究动态稀疏训练，论文里引用的稀疏方法有好几篇是他们自己的前作，这次他们也从稀疏下手。\n\n## 定义问题\n\n### 31 之后\n\n作者先用现成的两种做法把 ConvNeXt-T 的卷积核往上加，每组只训练 120 个 epoch，先看趋势：\n\n| 最大卷积核 | 直接放大 | 用 RepLKNet 的做法 |\n| --- | ---: | ---: |\n| 7×7（原版 ConvNeXt） | 81.0% | — |\n| 31×31 | 80.5% | **81.5%** |\n| 51×51 | 80.4% | 81.3% |\n| 61×61 | 80.3% | 81.3% |\n\n*表：摘自原论文 Table 1，ConvNeXt-T 在 ImageNet-1K 上的 top-1 准确率。四个阶段的卷积核依次缩小，例如 51 那一行是 [51, 49, 47, 13]。*\n\n直接放大的那一列，卷积核越大越差。RepLKNet 的做法在 31×31 救回来了，比原版多 0.5 个百分点，可是再往上就不动了。而且 51×51 的版本要 9.3 GFLOPs，是原版 ConvNeXt-T 的两倍，61×61 更是到了 11.5 GFLOPs。\n\n作者的解释和局部性有关。ConvNeXt 的 stem 是步长 4 的 4×4 卷积，224×224 的输入进来以后只剩 56×56；在这样的特征图上用 51×51 的卷积核，几乎就是全局卷积了。ViT 那边也有类似的经验，Swin 这类局部窗口注意力通常比 ViT 的全局注意力好。卷积核大到盖住整张图，也就失去了只看附近的那份归纳偏置。\n\n所以要解决的有两件事。第一，卷积核要很大，同时还保有局部性；第二，计算量和参数量随卷积核边长的平方增长，这个成本要压下来。\n\n## 解决问题\n\n作者的配方分成两步。\n\n### 第一步：拆成两根长条\n\n把一个 M×M 的卷积核，换成两个平行的长条卷积，一个 M×N、一个 N×M，N 取 5。长边负责看远处，短边保留局部细节。RepLKNet 那个并排的 5×5 小卷积核也留着，三条分支各自接一个 BatchNorm 再相加。作者特别提到，每条分支都要先接 BatchNorm 再相加，这一步很关键。\n\n[![三栏示意图，比较三种大卷积核做法。左栏是 ConvNeXt 的 7×7 方形卷积核，每个通道 49 个权重。中栏是 RepLKNet 的 51×51 大方形，右边并排一个 5×5 小方形，共 2,626 个权重。右栏是 SLaK，并排画出一根 51×5 的竖条、一根 5×51 的横条和一个 5×5 小方形，共 535 个权重，三块上面散布着浅色的格子，代表设为 0 的权重，下方标注 40% 稀疏、宽度乘以 1.3](/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png)](/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png)\n\n*图：本文整理。三种做法每个通道要学的权重数；原论文采用 CC BY-NC-SA 4.0 许可，这里不转载原图。*\n\n这样拆能省多少，我们直接数一下。这里都是 depthwise 卷积，每个通道各有一组卷积核：\n\n```python\n# 每个通道一个卷积核的 depthwise 卷积：比较每个通道要学几个权重\nfor M in (7, 31, 51, 61):\n    dense = M * M                              # 一个 M×M 方形卷积核\n    rep = M * M + 5 * 5                        # RepLKNet：M×M 再并排一个 5×5\n    slak = M * 5 + 5 * M + 5 * 5               # SLaK：M×5、5×M 加上 5×5\n    print(f\"M={M:>2}：方形 {dense:>5}，RepLKNet {rep:>5}，拆成长条 {slak:>4}（方形的 {slak / dense:.0%}）\")\n```\n\n```\nM= 7：方形    49，RepLKNet    74，拆成长条   95（方形的 194%）\nM=31：方形   961，RepLKNet   986，拆成长条  335（方形的 35%）\nM=51：方形  2601，RepLKNet  2626，拆成长条  535（方形的 21%）\nM=61：方形  3721，RepLKNet  3746，拆成长条  635（方形的 17%）\n```\n\n卷积核小的时候拆开反而更贵，7×7 拆完是原来的将近两倍。到了 51×51，拆完只剩两成。方形的成本跟着 M² 增长，长条只跟着 M 增长，卷积核越大，差距越明显。\n\n拆成十字形之后，一层卷积看到的范围其实是个十字，四个角落空着。网络有很多层，叠起来就补上了。用 9×3 的小十字试试看：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\nM, N = 9, 3                                    # 用小一点的 9×3 演示，方便打印出来\nk = torch.zeros(M, M)\nc, h = M // 2, N // 2\nk[:, c - h:c + h + 1] = 1                      # M×N 的竖条\nk[c - h:c + h + 1, :] = 1                      # N×M 的横条\n\nprint(\"一层长条卷积能看到的位置（9×9 里有几格）：\", int(k.sum().item()))\nfor row in k.int().tolist():\n    print(\"\".join(\"■\" if v else \"·\" for v in row))\n\ntwo = F.conv2d(k[None, None], k[None, None], padding=M - 1)[0, 0] > 0   # 叠两层之后的范围\ncenter = two[c:c + M, c:c + M]                                          # 回到原本那块 9×9\nprint(f\"叠两层之后，原本 9×9 的范围里看得到 {int(center.sum().item())} 格；整个范围延伸到 {two.shape[0]}×{two.shape[1]}\")\n```\n\n```\n一层长条卷积能看到的位置（9×9 里有几格）： 45\n···■■■···\n···■■■···\n···■■■···\n■■■■■■■■■\n■■■■■■■■■\n■■■■■■■■■\n···■■■···\n···■■■···\n···■■■···\n叠两层之后，原本 9×9 的范围里看得到 81 格；整个范围延伸到 17×17\n```\n\n一层只看得到 81 格里的 45 格，叠两层就全满了。\n\n拆完之后，同样用 ConvNeXt-T 训练 120 个 epoch，最大卷积核 7、31、51、61 的准确率是 81.0%、81.3%、81.5%、81.4%，FLOPs 从 4.5G 只涨到 5.6G。51×51 第一次比 31×31 好，而且 5.4 GFLOPs 的成本，比 RepLKNet 做法的 9.3G 少了四成多。\n\n### 第二步：稀疏一点，宽一点\n\nResNeXt 提出「多分几组，把网络变宽」，ConvNeXt 把它推到极端，用 depthwise 卷积再加宽通道。这篇把同一句话改成稀疏的版本，作者写成「use sparse groups, expand more width」。\n\n\n> [!TIP]\n> \n> ResNeXt 的分组卷积可以看这篇：\n> \n> - **[[16.11] ResNeXt：探访下一个维度](/blog/papers/classic-cnns/resnext)**\n\n\n具体做法是让卷积的权重有 40% 为 0。各层的稀疏比例借用 SNIP 的结果，在整个网络里按 $|g \\cdot w|$，也就是梯度乘权重的大小，挑出最重要的那些权重，看每一层留下多少，就用那个比例；至于每一层里哪些位置是 0，一开始随机决定。训练时用动态稀疏，每隔一段时间剪掉数值最小的那些权重，再在随机的位置长回同样数量的权重。非零权重的总数不变，位置一直在调整。\n\n省下来的计算量，再拿去把每个阶段的通道数放大 1.3 倍：\n\n| 最大卷积核 | 拆成长条 | ＋40% 稀疏 | ＋宽度 1.3 倍 |\n| --- | ---: | ---: | ---: |\n| 7×7 | 81.0%（4.5G） | 80.0%（2.6G） | 81.1%（4.5G） |\n| 31×31 | 81.3%（5.0G） | 80.4%（2.9G） | 81.5%（4.8G） |\n| 51×51 | **81.5%**（5.4G） | **80.5%**（3.1G） | **81.6%**（5.0G） |\n| 61×61 | 81.4%（5.6G） | 80.4%（3.2G） | 81.5%（5.2G） |\n\n*表：摘自原论文 Table 2，ConvNeXt-T 训练 120 个 epoch 的 top-1 准确率，括号内是 FLOPs。*\n\n稀疏化让准确率掉了大约 1 个百分点，FLOPs 也少了 2G 左右；把宽度加回来之后，准确率又回到最高。61×61 这一行的 81.5%，和 RepLKNet 做法在 61×61 的 81.3% 相比，FLOPs 少了 55%。\n\n看 7×7 那一行。整套配方用在小卷积核上，从 81.0% 到 81.1%，几乎没变。换句话说，准确率是大卷积核带来的，拆长条和稀疏只是让大卷积核训练得动、算得起。\n\n标题说要超过 51×51，61×61 也确实试了，只是每一列都没比 51×51 好，所以最后的 SLaK 用的还是 51。\n\n### SLaK 的样子\n\nSLaK 直接沿用 ConvNeXt 的骨架。stem 是步长 4 的 4×4 卷积，四个阶段的块数，SLaK-T 是 [3, 3, 9, 3]，SLaK-S 和 SLaK-B 是 [3, 3, 27, 3]。四个阶段的卷积核依次是 [51, 49, 47, 13]，每一个都拆成 M×5、5×M 加上 5×5。整个网络稀疏 40%、宽度乘 1.3，所有大小的模型都用同一组设置，作者没有为各个模型另外调参数。\n\n## 讨论\n\n### ImageNet\n\n正式训练 300 个 epoch，和其他模型比：\n\n| 模型 | 参数量 | FLOPs | Top-1 |\n| --- | ---: | ---: | ---: |\n| Swin-T | 28M | 4.5G | 81.3% |\n| ConvNeXt-T | 29M | 4.5G | 82.1% |\n| **SLaK-T** | 30M（50M） | 5.0G（8.7G） | **82.5%** |\n| ConvNeXt-S | 50M | 8.7G | 83.1% |\n| **SLaK-S** | 55M（91M） | 9.8G（16.7G） | **83.8%** |\n| RepLKNet-31B | 79M | 15.3G | 83.5% |\n| ConvNeXt-B | 89M | 15.4G | 83.8% |\n| **SLaK-B** | 95M（158M） | 17.1G（28.5G） | **84.0%** |\n\n*表：摘自原论文 Table 3，224×224 输入。括号内是没有稀疏加速时的参数量与 FLOPs，原论文以蓝字标示。*\n\n每个大小都比 ConvNeXt 高，差距 0.2 到 0.7 个百分点。换成 384×384 输入，SLaK-B 是 85.5%，ConvNeXt-B 是 85.1%，差距变成 224 时的两倍。作者认为分辨率越高，大卷积核越有用武之地。\n\n不过括号里的数字值得多看一眼。没有稀疏加速的话，SLaK-T 是 50M 参数、8.7 GFLOPs，刚好就是 ConvNeXt-S 的大小，而 ConvNeXt-S 是 83.1%，比 SLaK-T 高。SLaK-S 的 91M、16.7G 也和 ConvNeXt-B 差不多，两者都是 83.8%。按没有加速的成本来比，SLaK 的优势就不见了。\n\n### 下游任务\n\n大卷积核真正吃香的地方在下游。三个任务都用 T 尺寸来比：\n\n| 任务 | ConvNeXt-T（7×7） | ConvNeXt-T（31×31，RepLKNet 做法） | SLaK-T（51×51） |\n| --- | ---: | ---: | ---: |\n| ADE20K 语义分割 mIoU | 44.6 | 45.0 | **46.2** |\n| PASCAL VOC 2007 检测 mAP | 80.6 | 81.3 | **82.7** |\n| COCO 检测 AP<sup>box</sup> | 47.3 | 47.8 | **48.4** |\n\n*表：整理自原论文 Table 4、5、6。ADE20K 用 UperNet，VOC 用 Faster R-CNN，COCO 用 Cascade Mask R-CNN；骨干预训练 120 个 epoch，COCO 微调 12 个 epoch。*\n\n卷积核越大，三个任务都越好。ADE20K 上 SLaK-T 比 ConvNeXt-T 多 1.6，比 ImageNet 上的差距大得多，甚至超过了 ConvNeXt-S 的 45.9。RepLKNet 也观察过同样的现象，分类分数差不多的模型，到了需要看大范围的下游任务上，可以差很多。\n\n换成完整训练，ADE20K 上 T、S、B 三个尺寸分别从 46.0、48.7、49.1 提升到 47.6、49.4、50.2；COCO 微调 36 个 epoch，AP<sup>box</sup> 从 50.4 到 51.3。\n\n作者也测了有效感受野。从验证集取 50 张图像放大到 1024×1024，算每个输入像素对最后一层中心点输出的贡献。ConvNeXt 的高贡献像素集中在正中间；31×31 的版本散得比较开，还是盖不满整张图；SLaK 散得最广，离中心很远的地方也出现了高贡献的点。\n\n### 实际跑起来\n\nFLOPs 是理论值，作者在附录 H 补了 A100 上的实测。用 FP32、没有任何稀疏加速，每秒能处理的图像数，ConvNeXt-T 是 1,779 张，Swin-T 是 1,305 张，SLaK-T 是 709 张。SLaK-T 的吞吐量只有 ConvNeXt-T 的四成，作者说考虑到网络宽了 1.3 倍、卷积核大了 7 倍以上，这个差距可以接受。\n\n再把配方一步步拆开，用 ConvNeXt-B 搭配 [51, 49, 47, 13] 的卷积核测吞吐量：\n\n| 做法 | 每秒图像数 |\n| --- | ---: |\n| 直接用 51×51 方形卷积核 | 96 |\n| 拆成长条 | 283 |\n| ＋40% 稀疏 | 282 |\n| ＋宽度 1.3 倍 | 198 |\n\n*表：摘自原论文 Table 12，A100、FP32、无稀疏加速。*\n\n拆长条是真的快，将近 3 倍。稀疏那一步从 283 到 282，一点也没变快，因为稀疏是用二值掩码实现的，GPU 照样把那些 0 乘一遍。可是加宽的成本是实打实的，吞吐量掉了三成。\n\n作者在局限性一节写得很坦白，论文里的推理 FLOPs 都是理论值，要等硬件支持非结构化稀疏才兑现得了。也就是说，论文主表里黑字的那些 FLOPs，是写给未来的 GPU 看的。\n\n## 结论\n\n把一个大方块拆成两根长条，是这篇最实用的一招。局部性留住了，成本只跟着边长线性增长，51×51 终于赢过 31×31，下游任务进步得更多。而且拆长条不挑硬件，自己的网络里马上就能试。\n\n稀疏那一半就可惜了。在一般 GPU 上，40% 的 0 照样要乘，加宽 1.3 倍的代价却是实实在在的，SLaK-T 的吞吐量只有 ConvNeXt-T 的四成。\n\n所以这款 Pro Max，长条那一半先带回家就好；稀疏那一半，等哪天 GPU 支持了再说吧。\n","imageSizes":{"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png":{"width":1640,"height":660}}}