{"notes":[{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"Kernel Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"en","path":"/content/papers/en/classic-cnns/slak.5a1c38b2740d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"beyond-31","text":"Beyond 31","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"step-1-split-it-into-two-strips","text":"Step 1: Split It into Two Strips","depth":3},{"id":"step-2-a-little-sparser-a-little-wider","text":"Step 2: A Little Sparser, a Little Wider","depth":3},{"id":"what-slak-looks-like","text":"What SLaK Looks Like","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"running-it-for-real","text":"Running It for Real","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"Making Convolutions Great Again","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"en","path":"/content/papers/en/classic-cnns/convnext.1a8312c7b164.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"redesigning-network-architecture","text":"Redesigning Network Architecture","depth":3},{"id":"introducing-modern-training-techniques","text":"Introducing Modern Training Techniques","depth":3},{"id":"macro-design","text":"Macro Design","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"large-kernel-convolution","text":"Large Kernel Convolution","depth":3},{"id":"micro-design","text":"Micro Design","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"summary-of-model-configurations","text":"Summary of Model Configurations","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"reparameterization/replknet","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"Giant Convolutional Kernels","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"en","path":"/content/papers/en/reparameterization/replknet.271ac18a9d7c.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"a-guide-to-using-large-kernels-in-convolution","text":"A Guide to Using Large Kernels in Convolution","depth":2},{"id":"deep-large-kernels-remain-efficient","text":"Deep Large Kernels Remain Efficient","depth":3},{"id":"large-kernels-heavily-rely-on-residual-connections","text":"Large Kernels Heavily Rely on Residual Connections","depth":3},{"id":"large-kernels-require-reparameterization","text":"Large Kernels Require Reparameterization","depth":3},{"id":"large-kernels-significantly-enhance-downstream-task-performance","text":"Large Kernels Significantly Enhance Downstream Task Performance","depth":3},{"id":"large-kernels-are-effective-even-with-small-feature-maps","text":"Large Kernels Are Effective Even with Small Feature Maps","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"evaluation-of-large-kernels","text":"Evaluation of Large Kernels","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"large-kernel-cnns-have-larger-erf-than-deep-small-kernel-models","text":"Large Kernel CNNs Have Larger ERF Than Deep Small Kernel Models","depth":3},{"id":"shape-bias","text":"Shape Bias","depth":3},{"id":"limitations","text":"Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]},{"id":"classic-cnns/resnext","category":"classic-cnns","legacyPath":"classic-cnns/1611-resnext","legacySlug":"1611-resnext","legacyPaths":[],"title":"ResNeXt","subtitle":"Exploring the Next Dimension","author":"Z. Yuan","paperDate":"2016-11-16","publishedOn":"2024-07-12","paperTitle":"Aggregated Residual Transformations for Deep Neural Networks","paperUrl":"https://arxiv.org/abs/1611.05431","bibliography":{"title":"Aggregated Residual Transformations for Deep Neural Networks","authors":["Saining Xie","Ross Girshick","Piotr Dollár","Zhuowen Tu","Kaiming He"],"url":"https://arxiv.org/abs/1611.05431","pdfUrl":"https://arxiv.org/pdf/1611.05431","codeUrl":"https://github.com/facebookresearch/ResNeXt","preprint":{"id":"1611.05431","submittedOn":"2016-11-16","url":"https://arxiv.org/abs/1611.05431"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1611.05431"},"citation":"@misc{classiccnnsresnext2016,\n  title = {{Aggregated Residual Transformations for Deep Neural Networks}},\n  author = {Saining Xie and Ross Girshick and Piotr Dollár and Zhuowen Tu and Kaiming He},\n  year = {2016},\n  eprint = {1611.05431},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1611.05431}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnext.4945bb6b3dae.bib"},"thumbnail":{"src":"/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg","width":1224,"height":576},"locale":"en","path":"/content/papers/en/classic-cnns/resnext.6dc90fbd49fc.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"generalization-to-downstream-tasks","text":"Generalization to Downstream Tasks","depth":3},{"id":"not-conducive-to-model-design","text":"Not Conducive to Model Design","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"model-configuration","text":"Model Configuration","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"training-performance-on-imagenet-1k","text":"Training Performance on ImageNet-1K","depth":3},{"id":"increasing-model-complexity","text":"Increasing Model Complexity","depth":3},{"id":"imagenet-1k-model-performance","text":"ImageNet-1K Model Performance","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"}]}],"text":"In the first half of 2022, convolutional networks put out two comeback papers in a row. ConvNeXt reworked ResNet step by step following Swin Transformer's design, enlarging the kernel from 3×3 to 7×7. Two months later, RepLKNet went bolder and used 31×31 outright, with reparameterized small kernels helping it train.\n\n\n> [!TIP]\n> \n> For these two, you can start with our notes:\n> \n> - **[[22.01] ConvNeXt: Making Convolutions Great Again](/blog/papers/classic-cnns/convnext)**\n> - **[[22.03] RepLKNet: Giant Convolutional Kernels](/blog/papers/reparameterization/replknet)**\n\n\nRepLKNet stopped at 31×31. This paper wants to know whether kernels can get bigger still, and whether accuracy keeps climbing at 51×51 and 61×61. The authors come from UT Austin, Eindhoven University of Technology, and other universities. Decebal Mocanu and first author Shiwei Liu have worked on dynamic sparse training for years, and several of the sparse methods the paper cites are their own earlier work. This time, too, they come at the problem through sparsity.\n\n## Defining the Problem\n\n### Beyond 31\n\nThe authors first scale up ConvNeXt-T's kernels with the two existing approaches, training each setting for only 120 epochs to get a first look at the trend:\n\n| Largest kernel | Naive enlargement | With RepLKNet's approach |\n| --- | ---: | ---: |\n| 7×7 (original ConvNeXt) | 81.0% | — |\n| 31×31 | 80.5% | **81.5%** |\n| 51×51 | 80.4% | 81.3% |\n| 61×61 | 80.3% | 81.3% |\n\n*Table: Excerpted from Table 1 of the paper, top-1 accuracy of ConvNeXt-T on ImageNet-1K. Kernels shrink stage by stage across the four stages; the 51 row, for example, is [51, 49, 47, 13].*\n\nIn the naive-enlargement column, bigger kernels do worse. RepLKNet's approach rescues 31×31, 0.5 percentage points above the original, but beyond that it stops moving. On top of that, the 51×51 version needs 9.3 GFLOPs, twice the original ConvNeXt-T, and 61×61 goes up to 11.5 GFLOPs.\n\nThe authors' explanation has to do with locality. ConvNeXt's stem is a 4×4 convolution with stride 4, so a 224×224 input comes out as just 56×56; a 51×51 kernel on a feature map like that is almost a global convolution. The ViT side has seen something similar, where local-window attention like Swin's usually beats ViT's global attention. Once a kernel is big enough to cover the whole image, it loses the inductive bias of looking only at what's nearby.\n\nSo two things need solving. First, the kernel has to be very large while still keeping locality. Second, compute and parameter count grow with the square of the kernel's side length, and that cost has to come down.\n\n## Solving the Problem\n\nThe authors' recipe comes in two steps.\n\n### Step 1: Split It into Two Strips\n\nAn M×M kernel is replaced with two parallel strip convolutions, one M×N and one N×M, with N set to 5. The long side looks far away, and the short side keeps the local detail. RepLKNet's parallel 5×5 small kernel stays too, and each of the three branches gets its own BatchNorm before they're summed. The authors stress that putting BatchNorm on every branch before the sum is a crucial step.\n\n[![Three-column diagram comparing three large-kernel approaches. The left column is ConvNeXt's 7×7 square kernel, with 49 weights per channel. The middle column is RepLKNet's large 51×51 square with a small 5×5 square beside it, 2,626 weights in total. The right column is SLaK, showing side by side a 51×5 vertical strip, a 5×51 horizontal strip, and a small 5×5 square, 535 weights in total; light-colored cells scattered over the three pieces represent weights set to 0, and the label below reads 40% sparse, width × 1.3](/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png)](/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png)\n\n*Figure: Drawn for this note. The number of weights each approach learns per channel; the paper is licensed under CC BY-NC-SA 4.0, so the original figure is not reproduced here.*\n\nLet's count how much this split saves. These are all depthwise convolutions, with one kernel per channel:\n\n```python\n# Depthwise convolution with one kernel per channel: compare how many weights each channel learns\nfor M in (7, 31, 51, 61):\n    dense = M * M                              # one M×M square kernel\n    rep = M * M + 5 * 5                        # RepLKNet: M×M plus a parallel 5×5\n    slak = M * 5 + 5 * M + 5 * 5               # SLaK: M×5 and 5×M plus 5×5\n    print(f\"M={M:>2}: square {dense:>5}, RepLKNet {rep:>5}, strips {slak:>4} ({slak / dense:.0%} of square)\")\n```\n\n```\nM= 7: square    49, RepLKNet    74, strips   95 (194% of square)\nM=31: square   961, RepLKNet   986, strips  335 (35% of square)\nM=51: square  2601, RepLKNet  2626, strips  535 (21% of square)\nM=61: square  3721, RepLKNet  3746, strips  635 (17% of square)\n```\n\nWhen the kernel is small, splitting it actually costs more; 7×7 split up comes to nearly twice the original. At 51×51, the split leaves only a fifth. The square's cost grows with M², while the strips grow only with M, so the bigger the kernel, the wider the gap.\n\nAfter the split into a cross shape, what a single layer sees is in fact a cross, with the four corners left empty. A network has many layers, though, and stacking them fills the corners in. Let's try it with a small 9×3 cross:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\nM, N = 9, 3                                    # demonstrate with a smaller 9×3 so it's easy to print\nk = torch.zeros(M, M)\nc, h = M // 2, N // 2\nk[:, c - h:c + h + 1] = 1                      # the M×N vertical strip\nk[c - h:c + h + 1, :] = 1                      # the N×M horizontal strip\n\nprint(\"Positions one layer of strip convolution can see (cells out of 9×9):\", int(k.sum().item()))\nfor row in k.int().tolist():\n    print(\"\".join(\"■\" if v else \"·\" for v in row))\n\ntwo = F.conv2d(k[None, None], k[None, None], padding=M - 1)[0, 0] > 0   # coverage after stacking two layers\ncenter = two[c:c + M, c:c + M]                                          # back to the original 9×9 block\nprint(f\"After two layers, {int(center.sum().item())} cells of the original 9×9 are visible; the full range extends to {two.shape[0]}×{two.shape[1]}\")\n```\n\n```\nPositions one layer of strip convolution can see (cells out of 9×9): 45\n···■■■···\n···■■■···\n···■■■···\n■■■■■■■■■\n■■■■■■■■■\n■■■■■■■■■\n···■■■···\n···■■■···\n···■■■···\nAfter two layers, 81 cells of the original 9×9 are visible; the full range extends to 17×17\n```\n\nOne layer sees only 45 of the 81 cells; two layers fill them all.\n\nAfter the split, again training ConvNeXt-T for 120 epochs, largest kernels of 7, 31, 51, and 61 reach 81.0%, 81.3%, 81.5%, and 81.4%, while FLOPs grow only from 4.5G to 5.6G. For the first time, 51×51 beats 31×31, and its 5.4 GFLOPs cost over 40% less than the 9.3G of RepLKNet's approach.\n\n### Step 2: A Little Sparser, a Little Wider\n\nResNeXt proposed “split into more groups and make the network wider,” and ConvNeXt pushed that to the extreme with depthwise convolution plus wider channels. This paper turns the same line into a sparse version, which the authors write as “use sparse groups, expand more width.”\n\n\n> [!TIP]\n> \n> For ResNeXt's grouped convolution, see this note:\n> \n> - **[[16.11] ResNeXt: Exploring the Next Dimension](/blog/papers/classic-cnns/resnext)**\n\n\nConcretely, 40% of the convolution weights are set to 0. The per-layer sparsity ratios are borrowed from SNIP. Across the whole network, the most important weights are picked by $|g \\cdot w|$, the magnitude of gradient times weight, and however many each layer keeps becomes that layer's ratio; which positions within each layer are 0 is decided at random at the start. Training uses dynamic sparsity. Every so often, the smallest-magnitude weights are pruned and the same number of weights are regrown at random positions. The total number of nonzero weights stays fixed while their positions keep shifting.\n\nThe compute saved is then spent on widening every stage's channels by 1.3×:\n\n| Largest kernel | Split into strips | + 40% sparsity | + 1.3× width |\n| --- | ---: | ---: | ---: |\n| 7×7 | 81.0% (4.5G) | 80.0% (2.6G) | 81.1% (4.5G) |\n| 31×31 | 81.3% (5.0G) | 80.4% (2.9G) | 81.5% (4.8G) |\n| 51×51 | **81.5%** (5.4G) | **80.5%** (3.1G) | **81.6%** (5.0G) |\n| 61×61 | 81.4% (5.6G) | 80.4% (3.2G) | 81.5% (5.2G) |\n\n*Table: Excerpted from Table 2 of the paper, top-1 accuracy of ConvNeXt-T trained for 120 epochs, with FLOPs in parentheses.*\n\nSparsifying costs about 1 percentage point of accuracy and cuts about 2G of FLOPs; with the width added back, accuracy returns to the top. The 81.5% in the 61×61 row uses 55% fewer FLOPs than the 81.3% of RepLKNet's approach at 61×61.\n\nLook at the 7×7 row. Applied to a small kernel, the whole recipe goes from 81.0% to 81.1%, barely a change. In other words, the accuracy comes from the large kernel, and the strips and sparsity just make the large kernel trainable and affordable.\n\nThe title promises going beyond 51×51, and 61×61 was indeed tried, but it doesn't beat 51×51 in any column, so the final SLaK sticks with 51.\n\n### What SLaK Looks Like\n\nSLaK reuses ConvNeXt's skeleton directly. The stem is a 4×4 convolution with stride 4, and the block counts of the four stages are [3, 3, 9, 3] for SLaK-T and [3, 3, 27, 3] for SLaK-S and SLaK-B. The four stages' kernels are [51, 49, 47, 13] in order, each split into M×5 and 5×M plus 5×5. The whole network is 40% sparse with width multiplied by 1.3, and every model size uses this same setting; the authors didn't tune hyperparameters separately for each model.\n\n## Discussion\n\n### ImageNet\n\nWith the full 300-epoch training, compared against other models:\n\n| Model | Parameters | FLOPs | Top-1 |\n| --- | ---: | ---: | ---: |\n| Swin-T | 28M | 4.5G | 81.3% |\n| ConvNeXt-T | 29M | 4.5G | 82.1% |\n| **SLaK-T** | 30M (50M) | 5.0G (8.7G) | **82.5%** |\n| ConvNeXt-S | 50M | 8.7G | 83.1% |\n| **SLaK-S** | 55M (91M) | 9.8G (16.7G) | **83.8%** |\n| RepLKNet-31B | 79M | 15.3G | 83.5% |\n| ConvNeXt-B | 89M | 15.4G | 83.8% |\n| **SLaK-B** | 95M (158M) | 17.1G (28.5G) | **84.0%** |\n\n*Table: Excerpted from Table 3 of the paper, 224×224 input. The numbers in parentheses are parameter counts and FLOPs without sparse acceleration, shown in blue in the paper.*\n\nAt every size SLaK beats ConvNeXt, by 0.2 to 0.7 percentage points. With 384×384 input, SLaK-B reaches 85.5% and ConvNeXt-B 85.1%, twice the gap at 224. The authors believe large kernels have more room to shine at higher resolutions.\n\nThe numbers in parentheses deserve a second look, though. Without sparse acceleration, SLaK-T has 50M parameters and 8.7 GFLOPs, exactly the size of ConvNeXt-S, and ConvNeXt-S reaches 83.1%, higher than SLaK-T. SLaK-S's 91M and 16.7G are also about the same as ConvNeXt-B, and both sit at 83.8%. Compared at the unaccelerated cost, SLaK's advantage disappears.\n\n### Downstream Tasks\n\nDownstream is where large kernels really pay off. All three tasks compare the T size:\n\n| Task | ConvNeXt-T (7×7) | ConvNeXt-T (31×31, RepLKNet's approach) | SLaK-T (51×51) |\n| --- | ---: | ---: | ---: |\n| ADE20K semantic segmentation mIoU | 44.6 | 45.0 | **46.2** |\n| PASCAL VOC 2007 detection mAP | 80.6 | 81.3 | **82.7** |\n| COCO detection AP<sup>box</sup> | 47.3 | 47.8 | **48.4** |\n\n*Table: Compiled from Tables 4, 5, and 6 of the paper. ADE20K uses UperNet, VOC uses Faster R-CNN, and COCO uses Cascade Mask R-CNN; backbones are pretrained for 120 epochs, and COCO is fine-tuned for 12 epochs.*\n\nThe bigger the kernel, the better on all three tasks. On ADE20K, SLaK-T is 1.6 above ConvNeXt-T, a far bigger gap than on ImageNet, and it even passes ConvNeXt-S's 45.9. RepLKNet observed the same phenomenon, where models with similar classification scores can differ a lot on downstream tasks that need to see a wide area.\n\nWith full training, ADE20K results for the T, S, and B sizes improve from 46.0, 48.7, and 49.1 to 47.6, 49.4, and 50.2; with 36 epochs of COCO fine-tuning, AP<sup>box</sup> goes from 50.4 to 51.3.\n\nThe authors also measured the effective receptive field. They take 50 images from the validation set, resize them to 1024×1024, and compute how much each input pixel contributes to the center point of the last layer's output. ConvNeXt's high-contribution pixels cluster right in the middle. The 31×31 version spreads out more but still doesn't cover the whole image. SLaK spreads the widest, with high-contribution points showing up even far from the center.\n\n### Running It for Real\n\nFLOPs are theoretical, so in Appendix H the authors add measurements on an A100. In FP32 without any sparse acceleration, the number of images processed per second is 1,779 for ConvNeXt-T, 1,305 for Swin-T, and 709 for SLaK-T. SLaK-T's throughput is only 40% of ConvNeXt-T's; the authors say the gap is acceptable given that the network is 1.3× wider and the kernels are more than 7× larger.\n\nNext, they break the recipe down step by step and measure throughput with ConvNeXt-B and [51, 49, 47, 13] kernels:\n\n| Approach | Images per second |\n| --- | ---: |\n| Plain 51×51 square kernels | 96 |\n| Split into strips | 283 |\n| + 40% sparsity | 282 |\n| + 1.3× width | 198 |\n\n*Table: Excerpted from Table 12 of the paper, A100, FP32, no sparse acceleration.*\n\nSplitting into strips really is fast, nearly 3×. The sparsity step goes from 283 to 282, no faster at all, because the sparsity is implemented with binary masks and the GPU multiplies all those zeros anyway. The cost of widening, on the other hand, is very real, and throughput drops by 30%.\n\nIn the limitations section, the authors say candidly that the inference FLOPs in the paper are all theoretical and will only pay off once hardware supports unstructured sparsity. In other words, the FLOPs in black in the paper's main table are written for the GPUs of the future to read.\n\n## Conclusion\n\nSplitting one big square into two strips is the most practical trick in this paper. Locality is preserved, the cost grows only linearly with the side length, 51×51 finally beats 31×31, and downstream tasks improve even more. Splitting into strips also works on any hardware, so you can try it in your own network right away.\n\nThe sparse half is a letdown. On ordinary GPUs, the 40% zeros still get multiplied, while the 1.3× widening costs you for real, and SLaK-T's throughput is only 40% of ConvNeXt-T's.\n\nSo with this Pro Max, take the strip half home for now; the sparse half can wait until GPUs support it someday.\n","imageSizes":{"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png":{"width":1640,"height":660}}}