{"notes":[{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"卷積核 Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/slak.9c5ac51ae21b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"31-之後","text":"31 之後","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"第一步拆成兩根長條","text":"第一步：拆成兩根長條","depth":3},{"id":"第二步稀疏一點寬一點","text":"第二步：稀疏一點，寬一點","depth":3},{"id":"slak-的樣子","text":"SLaK 的樣子","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任務","text":"下游任務","depth":3},{"id":"實際跑起來","text":"實際跑起來","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"讓卷積再次偉大","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/convnext.667fd92d121a.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"重新設計網路架構","text":"重新設計網路架構","depth":3},{"id":"引入現代訓練技術","text":"引入現代訓練技術","depth":3},{"id":"宏觀設計","text":"宏觀設計","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"大核卷積","text":"大核卷積","depth":3},{"id":"微觀設計","text":"微觀設計","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"總結模型配置","text":"總結模型配置","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"用卷積混合影像切塊"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"用卷積混合影像切塊"},{"id":"reparameterization/replknet","relation":"comparison","topic":"大卷積核與簡化骨架"}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"巨無霸卷積核","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"zh-hant","path":"/content/papers/zh-hant/reparameterization/replknet.16fd94d59060.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"大核卷積使用說明書","text":"大核卷積使用說明書","depth":2},{"id":"深度大核卷積仍然高效","text":"深度大核卷積仍然高效","depth":3},{"id":"大核卷積十分依賴殘差連接","text":"大核卷積十分依賴殘差連接","depth":3},{"id":"大核卷積需要重新參數化","text":"大核卷積需要重新參數化","depth":3},{"id":"大核卷積明顯提升下游任務性能","text":"大核卷積明顯提升下游任務性能","depth":3},{"id":"大核卷積對於小特徵圖的效果依然很好","text":"大核卷積對於小特徵圖的效果依然很好","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"網路架構","text":"網路架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"大核卷積評估","text":"大核卷積評估","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"大核-cnn-比深度小核模型有更大的-erf","text":"大核 CNN 比深度小核模型有更大的 ERF","depth":3},{"id":"形狀偏差","text":"形狀偏差","depth":3},{"id":"限制","text":"限制","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"大卷積核與簡化骨架"}]},{"id":"classic-cnns/resnext","category":"classic-cnns","legacyPath":"classic-cnns/1611-resnext","legacySlug":"1611-resnext","legacyPaths":[],"title":"ResNeXt","subtitle":"探訪下一個維度","author":"Z. Yuan","paperDate":"2016-11-16","publishedOn":"2024-07-12","paperTitle":"Aggregated Residual Transformations for Deep Neural Networks","paperUrl":"https://arxiv.org/abs/1611.05431","bibliography":{"title":"Aggregated Residual Transformations for Deep Neural Networks","authors":["Saining Xie","Ross Girshick","Piotr Dollár","Zhuowen Tu","Kaiming He"],"url":"https://arxiv.org/abs/1611.05431","pdfUrl":"https://arxiv.org/pdf/1611.05431","codeUrl":"https://github.com/facebookresearch/ResNeXt","preprint":{"id":"1611.05431","submittedOn":"2016-11-16","url":"https://arxiv.org/abs/1611.05431"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1611.05431"},"citation":"@misc{classiccnnsresnext2016,\n  title = {{Aggregated Residual Transformations for Deep Neural Networks}},\n  author = {Saining Xie and Ross Girshick and Piotr Dollár and Zhuowen Tu and Kaiming He},\n  year = {2016},\n  eprint = {1611.05431},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1611.05431}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnext.4945bb6b3dae.bib"},"thumbnail":{"src":"/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg","width":1224,"height":576},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/resnext.6f895ad74065.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"下游任務泛化性","text":"下游任務泛化性","depth":3},{"id":"不利於模型設計","text":"不利於模型設計","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"網路架構","text":"網路架構","depth":3},{"id":"模型配置","text":"模型配置","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-1k-訓練效果","text":"ImageNet-1K 訓練效果","depth":3},{"id":"增加模型複雜度","text":"增加模型複雜度","depth":3},{"id":"imagenet-1k-模型表現","text":"ImageNet-1K 模型表現","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分組卷積與特徵重用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分組卷積與特徵重用"}]}],"text":"2022 年上半年，卷積網路連著出了兩篇翻身之作。ConvNeXt 照著 Swin Transformer 的設計一步步改 ResNet，卷積核從 3×3 放大到 7×7；兩個月後 RepLKNet 更大膽，直接用 31×31，再靠重參數化的小卷積核幫忙訓練。\n\n\n> [!TIP]\n> \n> 這兩篇可以先看我們的筆記：\n> \n> - **[[22.01] ConvNeXt：讓卷積再次偉大](/blog/papers/classic-cnns/convnext)**\n> - **[[22.03] RepLKNet：巨無霸卷積核](/blog/papers/reparameterization/replknet)**\n\n\nRepLKNet 停在 31×31。這篇論文想知道卷積核還能不能更大，放到 51×51、61×61 時準確率會不會繼續往上。作者群來自 UT Austin、Eindhoven 理工大學等學校。Decebal Mocanu 和第一作者 Shiwei Liu 長年研究動態稀疏訓練，論文裡引用的稀疏方法有好幾篇是他們自己的前作，這次他們也從稀疏下手。\n\n## 定義問題\n\n### 31 之後\n\n作者先用現成的兩種做法把 ConvNeXt-T 的卷積核往上加，每組只訓練 120 個 epoch，先看趨勢：\n\n| 最大卷積核 | 直接放大 | 用 RepLKNet 的做法 |\n| --- | ---: | ---: |\n| 7×7（原版 ConvNeXt） | 81.0% | — |\n| 31×31 | 80.5% | **81.5%** |\n| 51×51 | 80.4% | 81.3% |\n| 61×61 | 80.3% | 81.3% |\n\n*表：摘自原論文 Table 1，ConvNeXt-T 在 ImageNet-1K 的 top-1 準確率。四個階段的卷積核依序縮小，例如 51 那一列是 [51, 49, 47, 13]。*\n\n直接放大的那一欄，卷積核越大越差。RepLKNet 的做法在 31×31 救回來了，比原版多 0.5 個百分點，可是再往上就不動了。而且 51×51 的版本要 9.3 GFLOPs，是原版 ConvNeXt-T 的兩倍，61×61 更到 11.5 GFLOPs。\n\n作者的解釋和局部性有關。ConvNeXt 的 stem 是步長 4 的 4×4 卷積，224×224 的輸入進來以後只剩 56×56；在這樣的特徵圖上用 51×51 的卷積核，幾乎就是全域卷積了。ViT 那邊有類似的經驗，Swin 這類局部視窗注意力通常比 ViT 的全域注意力好。卷積核大到蓋住整張圖，也就失去了只看附近的那份歸納偏置。\n\n所以要解決的有兩件事。第一，卷積核要很大，同時還保有局部性；第二，計算量和參數量隨卷積核邊長的平方成長，這個成本要壓下來。\n\n## 解決問題\n\n作者的配方分成兩步。\n\n### 第一步：拆成兩根長條\n\n把一個 M×M 的卷積核，換成兩個平行的長條卷積，一個 M×N、一個 N×M，N 取 5。長邊負責看遠處，短邊保留局部細節。RepLKNet 那個並排的 5×5 小卷積核也留著，三條分支各自接一個 BatchNorm 再相加。作者特別提到，每條分支都要先接 BatchNorm 再相加，這一步很關鍵。\n\n[![三欄示意圖，比較三種大卷積核做法。左欄是 ConvNeXt 的 7×7 方形卷積核，每個通道 49 個權重。中欄是 RepLKNet 的 51×51 大方形，右邊並排一個 5×5 小方形，共 2,626 個權重。右欄是 SLaK，並排畫出一根 51×5 的直條、一根 5×51 的橫條和一個 5×5 小方形，共 535 個權重，三塊上面散布著淡色的格子，代表設為 0 的權重，下方標註 40% 稀疏、寬度乘以 1.3](/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png)](/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png)\n\n*圖：本文整理。三種做法每個通道要學的權重數；原論文以 CC BY-NC-SA 4.0 授權，這裡不轉載原圖。*\n\n這樣拆能省多少，我們直接數一下。這裡都是 depthwise 卷積，每個通道各有一組卷積核：\n\n```python\n# 每個通道一個卷積核的 depthwise 卷積：比較每個通道要學幾個權重\nfor M in (7, 31, 51, 61):\n    dense = M * M                              # 一個 M×M 方形卷積核\n    rep = M * M + 5 * 5                        # RepLKNet：M×M 再並排一個 5×5\n    slak = M * 5 + 5 * M + 5 * 5               # SLaK：M×5、5×M 加上 5×5\n    print(f\"M={M:>2}：方形 {dense:>5}，RepLKNet {rep:>5}，拆成長條 {slak:>4}（方形的 {slak / dense:.0%}）\")\n```\n\n```\nM= 7：方形    49，RepLKNet    74，拆成長條   95（方形的 194%）\nM=31：方形   961，RepLKNet   986，拆成長條  335（方形的 35%）\nM=51：方形  2601，RepLKNet  2626，拆成長條  535（方形的 21%）\nM=61：方形  3721，RepLKNet  3746，拆成長條  635（方形的 17%）\n```\n\n卷積核小的時候拆開反而更貴，7×7 拆完是原本的將近兩倍。到了 51×51，拆完只剩兩成。方形的成本跟著 M² 長，長條只跟著 M 長，卷積核越大，差距越明顯。\n\n拆成十字形之後，一層卷積看到的範圍其實是個十字，四個角落空著。網路有很多層，疊起來就補上了。用 9×3 的小十字試試看：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\nM, N = 9, 3                                    # 用小一點的 9×3 示範，方便印出來\nk = torch.zeros(M, M)\nc, h = M // 2, N // 2\nk[:, c - h:c + h + 1] = 1                      # M×N 的直條\nk[c - h:c + h + 1, :] = 1                      # N×M 的橫條\n\nprint(\"一層長條卷積能看到的位置（9×9 裡有幾格）：\", int(k.sum().item()))\nfor row in k.int().tolist():\n    print(\"\".join(\"■\" if v else \"·\" for v in row))\n\ntwo = F.conv2d(k[None, None], k[None, None], padding=M - 1)[0, 0] > 0   # 疊兩層之後的範圍\ncenter = two[c:c + M, c:c + M]                                          # 回到原本那塊 9×9\nprint(f\"疊兩層之後，原本 9×9 的範圍裡看得到 {int(center.sum().item())} 格；整個範圍延伸到 {two.shape[0]}×{two.shape[1]}\")\n```\n\n```\n一層長條卷積能看到的位置（9×9 裡有幾格）： 45\n···■■■···\n···■■■···\n···■■■···\n■■■■■■■■■\n■■■■■■■■■\n■■■■■■■■■\n···■■■···\n···■■■···\n···■■■···\n疊兩層之後，原本 9×9 的範圍裡看得到 81 格；整個範圍延伸到 17×17\n```\n\n一層只看得到 81 格裡的 45 格，疊兩層就全滿了。\n\n拆完之後，同樣用 ConvNeXt-T 訓練 120 個 epoch，最大卷積核 7、31、51、61 的準確率是 81.0%、81.3%、81.5%、81.4%，FLOPs 從 4.5G 只長到 5.6G。51×51 第一次比 31×31 好，而且 5.4 GFLOPs 的成本，比 RepLKNet 做法的 9.3G 少了四成多。\n\n### 第二步：稀疏一點，寬一點\n\nResNeXt 提出「多分幾組，把網路變寬」，ConvNeXt 把它推到極端，用 depthwise 卷積再加寬通道。這篇把同一句話改成稀疏的版本，作者寫成「use sparse groups, expand more width」。\n\n\n> [!TIP]\n> \n> ResNeXt 的分組卷積可以看這篇：\n> \n> - **[[16.11] ResNeXt：探訪下一個維度](/blog/papers/classic-cnns/resnext)**\n\n\n具體做法是讓卷積的權重有 40% 為 0。各層的稀疏比例借用 SNIP 的結果，在整個網路裡依 $|g \\cdot w|$，也就是梯度乘權重的大小，挑出最重要的那些權重，看每一層留下多少，就用那個比例；至於每一層裡哪些位置是 0，一開始隨機決定。訓練時用動態稀疏，每隔一段時間剪掉數值最小的那些權重，再在隨機的位置長回同樣數量的權重。非零權重的總數不變，位置一直在調整。\n\n省下來的計算量，再拿去把每個階段的通道數放大 1.3 倍：\n\n| 最大卷積核 | 拆成長條 | ＋40% 稀疏 | ＋寬度 1.3 倍 |\n| --- | ---: | ---: | ---: |\n| 7×7 | 81.0%（4.5G） | 80.0%（2.6G） | 81.1%（4.5G） |\n| 31×31 | 81.3%（5.0G） | 80.4%（2.9G） | 81.5%（4.8G） |\n| 51×51 | **81.5%**（5.4G） | **80.5%**（3.1G） | **81.6%**（5.0G） |\n| 61×61 | 81.4%（5.6G） | 80.4%（3.2G） | 81.5%（5.2G） |\n\n*表：摘自原論文 Table 2，ConvNeXt-T 訓練 120 個 epoch 的 top-1 準確率，括號內是 FLOPs。*\n\n稀疏化讓準確率掉了大約 1 個百分點，FLOPs 也少了 2G 左右；把寬度加回來之後，準確率又回到最高。61×61 這一列的 81.5%，和 RepLKNet 做法在 61×61 的 81.3% 相比，FLOPs 少了 55%。\n\n看 7×7 那一列。整套配方用在小卷積核上，從 81.0% 到 81.1%，幾乎沒變。換句話說，準確率是大卷積核帶來的，拆長條和稀疏只是讓大卷積核訓練得動、算得起。\n\n標題說要超過 51×51，61×61 也確實試了，只是每一欄都沒比 51×51 好，所以最後的 SLaK 用的還是 51。\n\n### SLaK 的樣子\n\nSLaK 直接沿用 ConvNeXt 的骨架。stem 是步長 4 的 4×4 卷積，四個階段的區塊數，SLaK-T 是 [3, 3, 9, 3]，SLaK-S 和 SLaK-B 是 [3, 3, 27, 3]。四個階段的卷積核依序是 [51, 49, 47, 13]，每一個都拆成 M×5、5×M 加上 5×5。整個網路稀疏 40%、寬度乘 1.3，所有大小的模型都用同一組設定，作者沒有替各個模型另外調參數。\n\n## 討論\n\n### ImageNet\n\n正式訓練 300 個 epoch，和其他模型比：\n\n| 模型 | 參數量 | FLOPs | Top-1 |\n| --- | ---: | ---: | ---: |\n| Swin-T | 28M | 4.5G | 81.3% |\n| ConvNeXt-T | 29M | 4.5G | 82.1% |\n| **SLaK-T** | 30M（50M） | 5.0G（8.7G） | **82.5%** |\n| ConvNeXt-S | 50M | 8.7G | 83.1% |\n| **SLaK-S** | 55M（91M） | 9.8G（16.7G） | **83.8%** |\n| RepLKNet-31B | 79M | 15.3G | 83.5% |\n| ConvNeXt-B | 89M | 15.4G | 83.8% |\n| **SLaK-B** | 95M（158M） | 17.1G（28.5G） | **84.0%** |\n\n*表：摘自原論文 Table 3，224×224 輸入。括號內是沒有稀疏加速時的參數量與 FLOPs，原論文以藍字標示。*\n\n每個大小都比 ConvNeXt 高，差距 0.2 到 0.7 個百分點。換成 384×384 輸入，SLaK-B 是 85.5%，ConvNeXt-B 是 85.1%，差距變成 224 時的兩倍。作者認為解析度越高，大卷積核越有用武之地。\n\n不過括號裡的數字值得多看一眼。沒有稀疏加速的話，SLaK-T 是 50M 參數、8.7 GFLOPs，剛好就是 ConvNeXt-S 的大小，而 ConvNeXt-S 是 83.1%，比 SLaK-T 高。SLaK-S 的 91M、16.7G 也和 ConvNeXt-B 差不多，兩者都是 83.8%。照沒有加速的成本來比，SLaK 的優勢就不見了。\n\n### 下游任務\n\n大卷積核真正吃香的地方在下游。三個任務都用 T 尺寸比：\n\n| 任務 | ConvNeXt-T（7×7） | ConvNeXt-T（31×31，RepLKNet 做法） | SLaK-T（51×51） |\n| --- | ---: | ---: | ---: |\n| ADE20K 語意分割 mIoU | 44.6 | 45.0 | **46.2** |\n| PASCAL VOC 2007 偵測 mAP | 80.6 | 81.3 | **82.7** |\n| COCO 偵測 AP<sup>box</sup> | 47.3 | 47.8 | **48.4** |\n\n*表：整理自原論文 Table 4、5、6。ADE20K 用 UperNet，VOC 用 Faster R-CNN，COCO 用 Cascade Mask R-CNN；骨幹預訓練 120 個 epoch，COCO 微調 12 個 epoch。*\n\n卷積核越大，三個任務都越好。ADE20K 上 SLaK-T 比 ConvNeXt-T 多 1.6，比 ImageNet 上的差距大得多，甚至超過 ConvNeXt-S 的 45.9。RepLKNet 也觀察過同樣的現象，分類分數差不多的模型，到了需要看大範圍的下游任務可以差很多。\n\n換成完整訓練，ADE20K 上 T、S、B 三個尺寸分別從 46.0、48.7、49.1 進步到 47.6、49.4、50.2；COCO 微調 36 個 epoch，AP<sup>box</sup> 從 50.4 到 51.3。\n\n作者也量了有效感受野。從驗證集取 50 張影像放大到 1024×1024，算每個輸入像素對最後一層中心點輸出的貢獻。ConvNeXt 的高貢獻像素集中在正中間；31×31 的版本散得比較開，還是蓋不滿整張圖；SLaK 散得最廣，離中心很遠的地方也出現了高貢獻的點。\n\n### 實際跑起來\n\nFLOPs 是理論值，作者在附錄 H 補了 A100 上的實測。用 FP32、沒有任何稀疏加速，每秒能處理的影像數，ConvNeXt-T 是 1,779 張，Swin-T 是 1,305 張，SLaK-T 是 709 張。SLaK-T 的吞吐量只有 ConvNeXt-T 的四成，作者說考慮到網路寬了 1.3 倍、卷積核大了 7 倍以上，這個差距可以接受。\n\n再把配方一步步拆開，用 ConvNeXt-B 搭 [51, 49, 47, 13] 的卷積核量吞吐量：\n\n| 做法 | 每秒影像數 |\n| --- | ---: |\n| 直接用 51×51 方形卷積核 | 96 |\n| 拆成長條 | 283 |\n| ＋40% 稀疏 | 282 |\n| ＋寬度 1.3 倍 | 198 |\n\n*表：摘自原論文 Table 12，A100、FP32、無稀疏加速。*\n\n拆長條是真的快，將近 3 倍。稀疏那一步從 283 到 282，一點也沒變快，因為稀疏是用二元遮罩實作的，GPU 照樣把那些 0 乘一遍。可是加寬的成本是實打實的，吞吐量掉了三成。\n\n作者在限制一節寫得很坦白，論文裡的推論 FLOPs 都是理論值，要等硬體支援非結構化稀疏才兌現得了。也就是說，論文主表裡黑字的那些 FLOPs，是寫給未來的 GPU 看的。\n\n## 結論\n\n把一個大方塊拆成兩根長條，是這篇最實用的一招。局部性留住了，成本只跟著邊長線性成長，51×51 終於贏過 31×31，下游任務進步得更多。而且拆長條不挑硬體，自己的網路裡馬上就能試。\n\n稀疏那一半就可惜了。在一般 GPU 上，40% 的 0 照樣要乘，加寬 1.3 倍的代價卻是實實在在的，SLaK-T 的吞吐量只有 ConvNeXt-T 的四成。\n\n所以這款 Pro Max，長條那一半先帶回家就好；稀疏那一半，等哪天 GPU 支援了再說吧。\n","imageSizes":{"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png":{"width":1640,"height":660}}}