{"notes":[{"id":"reparameterization/unireplknet","category":"reparameterization","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"UniRepLKNet","subtitle":"Large Kernels Cross Boundaries","author":"Z. Yuan","paperDate":"2023-11-27","publishedOn":"2025-12-14","world":"external/unireplknet","paperTitle":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","paperUrl":"https://arxiv.org/abs/2311.15599","bibliography":{"title":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","authors":["Xiaohan Ding","Yiyuan Zhang","Yixiao Ge","Sijie Zhao","Lin Song","Xiangyu Yue","Ying Shan"],"url":"https://arxiv.org/abs/2311.15599","pdfUrl":"https://arxiv.org/pdf/2311.15599v2","codeUrl":"https://github.com/AILab-CVC/UniRepLKNet","preprint":{"id":"2311.15599","submittedOn":"2023-11-27","url":"https://arxiv.org/abs/2311.15599"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Ding_UniRepLKNet_A_Universal_Perception_Large-Kernel_ConvNet_for_Audio_Video_Point_CVPR_2024_paper.html"},"citation":"@misc{reparameterizationunireplknet2023,\n  title = {{UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition}},\n  author = {Xiaohan Ding and Yiyuan Zhang and Yixiao Ge and Sijie Zhao and Lin Song and Xiangyu Yue and Ying Shan},\n  year = {2023},\n  eprint = {2311.15599},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2311.15599}\n}\n","citationPath":"/content/papers/citations/reparameterization/unireplknet.df5ca5b9a0be.bib"},"thumbnail":{"src":"/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png","width":1680,"height":384},"locale":"en","path":"/content/papers/en/reparameterization/unireplknet.136ac87079c1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"one-33-layer-does-three-jobs","text":"One 3×3 Layer Does Three Jobs","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"baseline-architecture","text":"Baseline Architecture","depth":3},{"id":"guideline-1-deepen-with-cheap-structures","text":"Guideline 1: Deepen with Cheap Structures","depth":3},{"id":"guideline-2-reparameterize-with-small-dilated-convolutions","text":"Guideline 2: Reparameterize with Small Dilated Convolutions","depth":3},{"id":"guideline-3-choose-kernel-size-for-the-downstream-task","text":"Guideline 3: Choose Kernel Size for the Downstream Task","depth":3},{"id":"guideline-4-add-small-kernels-when-adding-depth","text":"Guideline 4: Add Small Kernels When Adding Depth","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"images","text":"Images","depth":3},{"id":"other-modalities","text":"Other Modalities","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""},{"id":"lightweight/senet","relation":"referenced","topic":""}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"Giant Convolutional Kernels","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"en","path":"/content/papers/en/reparameterization/replknet.271ac18a9d7c.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"a-guide-to-using-large-kernels-in-convolution","text":"A Guide to Using Large Kernels in Convolution","depth":2},{"id":"deep-large-kernels-remain-efficient","text":"Deep Large Kernels Remain Efficient","depth":3},{"id":"large-kernels-heavily-rely-on-residual-connections","text":"Large Kernels Heavily Rely on Residual Connections","depth":3},{"id":"large-kernels-require-reparameterization","text":"Large Kernels Require Reparameterization","depth":3},{"id":"large-kernels-significantly-enhance-downstream-task-performance","text":"Large Kernels Significantly Enhance Downstream Task Performance","depth":3},{"id":"large-kernels-are-effective-even-with-small-feature-maps","text":"Large Kernels Are Effective Even with Small Feature Maps","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"evaluation-of-large-kernels","text":"Evaluation of Large Kernels","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"large-kernel-cnns-have-larger-erf-than-deep-small-kernel-models","text":"Large Kernel CNNs Have Larger ERF Than Deep Small Kernel Models","depth":3},{"id":"shape-bias","text":"Shape Bias","depth":3},{"id":"limitations","text":"Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]},{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"Kernel Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"en","path":"/content/papers/en/classic-cnns/slak.5a1c38b2740d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"beyond-31","text":"Beyond 31","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"step-1-split-it-into-two-strips","text":"Step 1: Split It into Two Strips","depth":3},{"id":"step-2-a-little-sparser-a-little-wider","text":"Step 2: A Little Sparser, a Little Wider","depth":3},{"id":"what-slak-looks-like","text":"What SLaK Looks Like","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"running-it-for-real","text":"Running It for Real","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]},{"id":"lightweight/senet","category":"lightweight","legacyPath":"lightweight/1709-senet","legacySlug":"1709-senet","legacyPaths":[],"title":"SENet","subtitle":"Squeeze and Excitation Network","author":"Z. Yuan","paperDate":"2017-09-05","publishedOn":"2024-06-28","paperTitle":"Squeeze-and-Excitation Networks","paperUrl":"https://arxiv.org/abs/1709.01507","bibliography":{"title":"Squeeze-and-Excitation Networks","authors":["Jie Hu","Li Shen","Samuel Albanie","Gang Sun","Enhua Wu"],"url":"https://arxiv.org/abs/1709.01507","pdfUrl":"https://arxiv.org/pdf/1709.01507","codeUrl":null,"preprint":{"id":"1709.01507","submittedOn":"2017-09-05","url":"https://arxiv.org/abs/1709.01507"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1709.01507"},"citation":"@misc{lightweightsenet2017,\n  title = {{Squeeze-and-Excitation Networks}},\n  author = {Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},\n  year = {2017},\n  eprint = {1709.01507},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1709.01507}\n}\n","citationPath":"/content/papers/citations/lightweight/senet.1766a7978dde.bib"},"thumbnail":{"src":"/content/papers/assets/322f16ed8c782ffbdd7d7db3e8292c991908e3543dc69e1addd9238e22484363.jpg","width":1688,"height":394},"locale":"en","path":"/content/papers/en/lightweight/senet.20270339ffe6.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"image-classification-results","text":"Image Classification Results","depth":3},{"id":"results-on-lightweight-models","text":"Results on Lightweight Models","depth":3},{"id":"importance-of-squeeze","text":"Importance of Squeeze","depth":3},{"id":"importance-of-excitation","text":"Importance of Excitation","depth":3},{"id":"other-experimental-results","text":"Other Experimental Results","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v3","relation":"comparison","topic":"Channel attention in lightweight networks"},{"id":"lightweight/pp-lcnet","relation":"comparison","topic":"Channel attention in lightweight networks"}]}],"text":"After RepLKNet expanded convolution kernels to 31×31, large kernels became a hot topic. SLaK split them into bands to reach 51×51, and others brought large kernels to detection, segmentation, and other tasks.\n\nFirst author Xiaohan Ding also authored RepLKNet and RepVGG. Now at Tencent AI Lab, he and researchers from the Chinese University of Hong Kong revisit the architecture of large-kernel networks. They find that most existing designs simply inherit someone else's architecture: RepLKNet follows Swin Transformer, SLaK follows ConvNeXt, and ConvNeXt itself uses medium-sized 7×7 kernels. Architectures designed specifically for large kernels remain little explored.\n\nThe paper asks another question too. Transformers have established themselves in images, speech, point clouds, and time series. Can convolutional networks compete beyond images?\n\n\n> [!TIP]\n> \n> The first two stops along the large-kernel route:\n> \n> - **[[22.03] RepLKNet: Giant Convolutional Kernels](/blog/papers/reparameterization/replknet)**\n> - **[[22.07] SLaK: Kernel Pro Max](/blog/papers/classic-cnns/slak)**\n\n\n## Defining the Problem\n\n### One 3×3 Layer Does Three Jobs\n\nThe authors start with an old issue in small-kernel networks. Adding a layer to a network built from 3×3 convolutions does three things at once:\n\n1. Expands the receptive field\n2. Raises the abstraction level of spatial features, for example from corners and textures to object shapes\n3. Deepens the network, adding parameters and nonlinearities to increase expressive capacity\n\nSmall-kernel networks have to stack layers to enlarge their receptive fields, tying these three jobs together. Large kernels can see widely without greater depth. A few large-kernel layers already give a sufficiently large effective receptive field, leaving compute for more efficient structures to handle the second and third jobs.\n\nThe authors argue that large-kernel networks should design these three jobs separately. The next four guidelines all follow from this starting point.\n\n## Solving the Problem\n\n### Baseline Architecture\n\n[![Figure 1 from the paper: the left shows a four-stage backbone, with 2× downsampling between stages 1 through 4 and 4× downsampling at the input. The middle expands stage 3, alternating orange LarK Blocks and blue SmaK Blocks, with two SmaK Blocks after each LarK Block. The dashed box on the right shows a LarK Block: depthwise convolution implemented as a Dilated Reparam Block, BN, an SE Block with average pooling, ReLU, and sigmoid multiplication, FFN, BN, then drop path and residual addition.](/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png)](/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png)\n\n*Figure 1 from Ding et al., taken from [**PDF page 2**](https://arxiv.org/pdf/2311.15599v2#page=2). The original caption is omitted.*\n\nStart with the experimental baseline. It has four stages with $C$, $2C$, $4C$, and $8C$ channels and $(3, 3, 9, 3)$ blocks, just like ConvNeXt-T. Each block resembles ConvNeXt: a depthwise convolution followed by an FFN with GRN. BatchNorm replaces LayerNorm so it can be merged into the convolution at inference. The first stage uses 3×3 kernels; the other three use 13×13.\n\nDuring architecture exploration, ImageNet training lasts only 100 epochs, followed by ADE20K segmentation with UperNet. The authors emphasize segmentation because large kernels' benefits are more apparent downstream; ImageNet accuracy alone can hide the difference.\n\n### Guideline 1: Deepen with Cheap Structures\n\nWith large kernels handling the receptive field, extra depth can come from structures that require no wide spatial computation. The authors add a small structure to each block, trying four options:\n\n| Added structure | Throughput (images/s) | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| None | 1954 | 81.2% | 45.1 |\n| Bottleneck (1×1, DW 3×3, 1×1) | 1716 | 81.5% | 46.3 |\n| Two 1×1 convolutions | 1745 | 81.3% | 46.2 |\n| Two DW 3×3 convolutions | 1659 | 81.3% | 45.4 |\n| SE Block | **1863** | **81.6%** | **46.5** |\n\n*Table: from the paper's Table 1. A100, batch size 128.*\n\nThe SE Block provides both channel interaction and spatial aggregation. Its spatial aggregation is global average pooling, which is cheap, giving the best result and the highest speed among the additions.\n\n\n> [!TIP]\n> \n> For how an SE Block uses global average pooling to weight channels, see:\n> \n> - **[[17.09] SENet: Squeeze and Excitation Network](/blog/papers/lightweight/senet)**\n\n\n### Guideline 2: Reparameterize with Small Dilated Convolutions\n\nRepLKNet already found that a large kernel benefits from a small parallel kernel during training to capture small-scale patterns. After training, BN is merged into the convolutions and the small kernel is zero-padded and added to the large one, leaving one layer for inference.\n\nThis paper takes that one step further. Beyond small-scale patterns, there are sparse patterns: a feature-map pixel may relate more strongly to pixels several positions away than to its immediate neighbors. Dilated convolution is well suited to this. At dilation $r$, adjacent sampling points have $r-1$ pixels between them.\n\n[![Figure 2 from the paper: the top shows the structural view. The input enters five parallel branches: 9×9 at dilation 1, 5×5 at dilation 1, 5×5 at dilation 2, 3×3 at dilation 3, and 3×3 at dilation 4. Each has BN, and their outputs are added. A Re-parameterize arrow points to a single 9×9 convolution at dilation 1. Below, the parameter view shows each branch's weights as grids; dilated weights expand into sparse grids with spaces between weights, then all grids overlap to form a 9×9 grid, with shades indicating summed weights from different branches.](/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png)](/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png)\n\n*Figure 2 from Ding et al., taken from [**PDF page 4**](https://arxiv.org/pdf/2311.15599v2#page=4). The original caption is omitted.*\n\nThe figure uses $K=9$ as an example; the actual network uses $K=13$. The lower half is the key: a $k\\times k$ convolution at dilation $r$ is equivalent to a sparse kernel of size $(k-1)r+1$ with zeros inserted between the weights. As a convolution kernel, it can be added to the large kernel.\n\nFirst, verify that a dilated convolution and its expanded sparse kernel compute exactly the same result:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ndef dilate(w, r):\n    \"\"\"Expand k×k weights by inserting r-1 zeros between neighboring weights.\"\"\"\n    c, _, k, _ = w.shape\n    size = (k - 1) * r + 1\n    big = torch.zeros(c, 1, size, size)\n    big[:, :, ::r, ::r] = w\n    return big\n\ntorch.manual_seed(0)\nc = 4\nx = torch.randn(1, c, 32, 32)\nw = torch.randn(c, 1, 3, 3)\ny_dilated = F.conv2d(x, w, padding=3, dilation=3, groups=c)   # 3×3 convolution at dilation 3\ny_sparse = F.conv2d(x, dilate(w, 3), padding=3, groups=c)     # Expanded 7×7 sparse kernel\nprint(\"Expanded kernel:\", tuple(dilate(w, 3).shape[-2:]), \"nonzero entries\", int((dilate(w, 3)[0] != 0).sum()))\nprint(\"Maximum difference between the two methods:\", f\"{(y_dilated - y_sparse).abs().max():.1e}\")\n```\n\n```text\nExpanded kernel: (7, 7) nonzero entries 9\nMaximum difference between the two methods: 0.0e+00\n```\n\nNow follow the paper's defaults. Beside the main 13×13 kernel are five parallel branches, with sizes $k=(5,7,3,3,3)$ and dilations $r=(1,2,3,4,5)$. During training, compute them separately and sum the outputs. Before inference, expand each branch, pad it to 13×13, and add it to the main kernel:\n\n```python\nK = 13\nbranches = [(5, 1), (7, 2), (3, 3), (3, 4), (3, 5)]          # (kernel size k, dilation r)\nw_main = torch.randn(c, 1, K, K)\nw_small = [torch.randn(c, 1, k, k) for k, _ in branches]\n\n# Training: one 13×13 plus five small dilated convolutions, computed separately and summed\ny_train = F.conv2d(x, w_main, padding=K // 2, groups=c)\nfor (k, r), w in zip(branches, w_small):\n    y_train += F.conv2d(x, w, padding=r * (k - 1) // 2, dilation=r, groups=c)\n\n# Before inference: expand and pad every branch to 13×13, then add to the main kernel\nmerged = w_main.clone()\nfor (k, r), w in zip(branches, w_small):\n    big = dilate(w, r)\n    pad = (K - big.shape[-1]) // 2\n    merged += F.pad(big, [pad] * 4)\n    print(f\"{k}×{k}, dilation {r} → equivalent {big.shape[-1]}×{big.shape[-1]}\")\ny_infer = F.conv2d(x, merged, padding=K // 2, groups=c)\nprint(\"Output magnitude:\", f\"{y_train.abs().max():.1f}\")\nprint(\"Maximum difference after merging into one 13×13:\", f\"{(y_train - y_infer).abs().max():.1e}\")\n```\n\n```text\n5×5, dilation 1 → equivalent 5×5\n7×7, dilation 2 → equivalent 13×13\n3×3, dilation 3 → equivalent 7×7\n3×3, dilation 4 → equivalent 9×9\n3×3, dilation 5 → equivalent 11×11\nOutput magnitude: 59.9\nMaximum difference after merging into one 13×13: 2.7e-05\n```\n\nSix convolutions merge into one, leaving only rounding error from the changed order of floating-point additions. In the actual module, each branch also has BN. Before inference, fold BN's scale and shift into the convolution weights and biases, just as in RepVGG.\n\n\n> [!TIP]\n> \n> For merging BN into convolutions and parallel branches into one path, see:\n> \n> - **[[21.01] RepVGG: Making VGG Great Again](/blog/papers/reparameterization/repvgg)**\n\n\nDoes this help? The authors compare two non-dilated variants with the same branch count: one keeps the same kernel sizes $(5,7,3,3,3)$; the other keeps the same effective sizes $(5,13,7,9,11)$. All three become the same 13×13 kernel at inference. Only training differs:\n\n| Reparameterization | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: |\n| None | 81.44 ± 0.04% | 45.78 ± 0.05 |\n| Small dilated convolutions | **81.63 ± 0.02%** | **46.37 ± 0.10** |\n| Same kernel sizes, no dilation | 81.55 ± 0.01% | 46.07 ± 0.07 |\n| Same effective sizes, no dilation | 81.59 ± 0.02% | 46.17 ± 0.04 |\n\n*Table: from the paper's Table 2; mean and standard deviation over three training runs.*\n\nThe dilated version performs best, adding 0.6 mIoU over no reparameterization. The authors interpret this as a benefit from dilated convolutions' ability to capture sparse patterns. Adding small kernels alone, or adding a range of receptive-field sizes, is less effective.\n\n### Guideline 3: Choose Kernel Size for the Downstream Task\n\nNext, how large should each stage's kernel be?\n\n| Kernels by stage | Throughput (images/s) | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 3, 13, 13, 13 (baseline) | 1863 | 81.6% | **46.5** |\n| 3, 11, 11, 11 | 1876 | 81.6% | 45.5 |\n| 3, 15, 15, 15 | 1851 | 81.7% | 45.9 |\n| 3, 3, 13, 13 | 2006 | 81.7% | 46.1 |\n| 13, 13, 13, 13 | 1547 | 81.6% | 44.9 |\n\n*Table: selected rows from the paper's Table 3.*\n\nImageNet accuracy barely changes, staying around 81.6%, while mIoU ranges from 44.9 to 46.5. Classification alone would suggest that kernel size matters little.\n\nStrangely, increasing kernels to 15, or also using 13 in the first stage, lowers mIoU. The authors suspect UperNet. It combines early and late backbone features, assuming early features carry local information. Large early kernels make those features less local, and the combination becomes less useful.\n\nTo test this, the authors let UperNet use only the fourth stage's output, evaluating the final features in isolation. Here, size 15 gives the highest mIoU, 42.7. Using 13 in the first stage gives 42.4, matching the baseline; size 11 is lowest at 41.9. Large kernels preserve feature quality, but make early features less suited to UperNet.\n\nKernel size therefore depends on the downstream component. For common image tasks, the authors default to 13×13 in the middle and higher stages.\n\n### Guideline 4: Add Small Kernels When Adding Depth\n\nFinally, scaling the model. The authors increase stage 3 from 9 to 27 blocks. Should the extra 18 also use large kernels? A block with a Dilated Reparam Block is called a LarK Block; one with DW 3×3 is a SmaK Block:\n\n| Stage 3 composition | Throughput (images/s) | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 9 LarK (original shallow variant) | 1863 | 81.6% | 46.5 |\n| 27 LarK | 1145 | 82.3% | **49.0** |\n| 14 LarK and 13 SmaK, alternating | 1229 | 82.3% | 48.8 |\n| 9 LarK and 18 SmaK | 1264 | 82.3% | 48.8 |\n| As above, removing DW 3×3 from SmaK | 1289 | 82.2% | 47.8 |\n\n*Table: from the paper's Table 4.*\n\nUsing large kernels throughout adds only 0.2 mIoU and costs nearly 10% in speed. Nine large-kernel blocks already sustain the receptive field; small kernels suffice for the extra depth.\n\nThe last row is revealing. Removing SmaK's 3×3 convolution brings only a small speed gain but loses one mIoU point. These small kernels contribute little to the receptive field, yet build higher levels of spatial abstraction—the second job discussed at the start.\n\nThe final UniRepLKNet follows this arrangement. From size S upward, stage 3 has 9 LarK and 18 SmaK blocks, with two SmaK blocks after each LarK, as shown in the architecture figure. The smallest T uses 9 plus 9.\n\n## Discussion\n\n### Images\n\nFirst, results trained only on ImageNet-1K. Throughput is measured on an A100 at batch size 128:\n\n| Model | Parameters | Throughput (images/s) | top-1 |\n| --- | ---: | ---: | ---: |\n| SLaK-T | 30M | 1312 | 82.5% |\n| InternImage-T | 30M | 1292 | 83.5% |\n| UniRepLKNet-T | 31M | **1804** | 83.2% |\n| ConvNeXt-S | 50M | 1182 | 83.1% |\n| UniRepLKNet-S | 56M | 1265 | **83.9%** |\n\n*Table: selected rows from the paper's Table 6.*\n\nUniRepLKNet-T scores 0.3 percentage points below InternImage-T but runs 40% faster. The largest variant, UniRepLKNet-XL, pretrained on ImageNet-22K at resolution 384, reaches 88.0%, matching InternImage-XL, with throughput of 131 versus 114.\n\nDownstream results are good too. On ADE20K, UniRepLKNet-T gets 48.6 mIoU with single-scale testing, against 46.0 for ConvNeXt-T; XL reaches 55.6 with multi-scale testing. On COCO, XL achieves 56.4 box AP.\n\n### Other Modalities\n\nTo use the same network on other data, the authors take a direct approach: arrange each kind of data as a $B\\times C'\\times H\\times W$ tensor. Change only the first layer's input channel count $C'$; everything else stays the same.\n\n- **Speech**: treat a spectrogram with $T$ time bins and $F$ frequency bins as a single-channel $T\\times F$ image.\n- **Time series**: use Corrformer's embedding layer to project into a latent space, then reshape into a single-channel two-dimensional map.\n- **Point clouds**: use a few convolution layers to produce projection maps from X, Y, and Z views, forming a three-channel 224×224 image.\n- **Video**: arrange 16 frames of 224×224 into a 4×4 grid, making one large 896×896 image.\n\nThe video approach is particularly blunt. Temporal relationships between frames rely entirely on kernels crossing tile boundaries.\n\nHere are the four modalities together:\n\n| Task | UniRepLKNet-S | Comparison |\n| --- | --- | --- |\n| Speech command recognition (Speech Commands V2) | 98.5%, 55.5M, no pretraining | Audio-MAE 98.3%, 86.2M, AudioSet-2M pretraining |\n| Global temperature prediction (MSE, lower is better) | 7.602 | Corrformer 7.709 |\n| 3D shape classification (ModelNet-40 OA) | 93.2% | DGCNN, KPConv 92.9% |\n| Action recognition (Kinetics-400) | 54.8%, no pretraining | MViTv2-B 81.2%, ImageNet-1K pretraining |\n\n*Table: selected rows from the paper's Tables 9 through 12.*\n\nSpeech, time series, and point clouds match or slightly outperform purpose-built models. Video lags far behind, and the authors acknowledge the gap to specialized video models. Their comparisons are with the likewise general-purpose ImageBind (50.0%) and Meta-Transformer (47.3%), both of which also use substantial pretraining data.\n\nKernel size matters here too. With the same preprocessing and training setup, substitute other convolutional networks:\n\n| Model (kernel size) | Speech | Video |\n| --- | ---: | ---: |\n| ResNet-101 (3) | 73.6% | 41.3% |\n| ConvNeXt-S (7) | 94.3% | 48.5% |\n| UniRepLKNet-S (11) | 94.7% | 51.7% |\n| UniRepLKNet-S (13) | **98.5%** | **54.8%** |\n\n*Table: selected columns from the paper's Table 13.*\n\nResNet-101 gets just 73.6% on speech; with a kernel of size 13, the result reaches 98.5%. Spectrograms and frame grids both need a wide view at once, exactly where large kernels help.\n\n### The Cost\n\nThe authors state one limitation in the conclusion: dilated branches require more training resources. Each of the five branches computes its own convolution and BN. They merge into one at inference, but that training work remains.\n\nInference speed also depends on the implementation. Large-kernel depthwise convolution needs the official accelerated implementation. When a user [reports slow training on GitHub](https://github.com/AILab-CVC/UniRepLKNet/issues/15), the author's first question is whether that implementation is installed. Another user [measures on a 4090](https://github.com/AILab-CVC/UniRepLKNet/issues/22) and finds UniRepLKNet-S no faster than ConvNeXt-B. The author replies that the paper measures on an A100; the speed ranking depends on hardware.\n\n## Conclusion\n\nUniRepLKNet's four guidelines share one idea. Large kernels provide the wide view, small kernels build levels of abstraction, and cheap structures such as SE add depth. Give each operation its own job instead of asking one operation to do all three.\n\nThe Dilated Reparam Block is the easiest piece to take away. Attach several small dilated branches during training, then merge them into a large kernel for inference. The deployment side sees the same operation. The code above contains all the kernel-merging logic; add BN fusion to complete it.\n\nThe multimodal experiments are more surprising. Treat spectrograms as images and arrange video frames in grids, add no specialized design, and a convolutional network can still match or slightly beat Transformers on several of their home grounds.\n","imageSizes":{"/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png":{"width":960,"height":621},"/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png":{"width":1680,"height":384}}}