{"notes":[{"id":"classic-cnns/wbmm","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"WBMM","subtitle":"Bigger Windows, Higher Speed","author":"Z. Yuan","paperDate":"2026-07-02","publishedOn":"2026-10-09","paperTitle":"WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution","paperUrl":"https://arxiv.org/abs/2607.02097","bibliography":{"title":"WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution","authors":["Wan Song","Wei Zhou","Rui Wang","Jun Yu","Toru Kurihara","Jiajia Xu","Shu Zhan"],"url":"https://arxiv.org/abs/2607.02097","pdfUrl":"https://arxiv.org/pdf/2607.02097v2","codeUrl":"https://github.com/wansong-s/WBMM","preprint":{"id":"2607.02097","submittedOn":"2026-07-02","url":"https://arxiv.org/abs/2607.02097"},"publication":{"venue":"ICML","year":2026,"kind":"conference","url":"https://proceedings.mlr.press/v306/song26ae.html"},"citation":"@misc{classiccnnswbmm2026,\n  title = {{WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution}},\n  author = {Wan Song and Wei Zhou and Rui Wang and Jun Yu and Toru Kurihara and Jiajia Xu and Shu Zhan},\n  year = {2026},\n  eprint = {2607.02097},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2607.02097}\n}\n","citationPath":"/content/papers/citations/classic-cnns/wbmm.5eb4a1ff4a34.bib"},"thumbnail":{"src":"/content/papers/assets/5c20ec217f297d24ad8ce71ca79936d0599f7b460167e3bb89f52515ff1347ee.png","width":1017,"height":360},"locale":"en","path":"/content/papers/en/classic-cnns/wbmm.2cf7c0224aa1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"convolution-waits-for-data","text":"Convolution Waits for Data","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"write-convolution-as-matrix-multiplication","text":"Write Convolution as Matrix Multiplication","depth":3},{"id":"bigger-windows-run-faster","text":"Bigger Windows Run Faster","depth":3},{"id":"let-windows-exchange-information","text":"Let Windows Exchange Information","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"detection-and-segmentation","text":"Detection and Segmentation","depth":3},{"id":"other-hardware","text":"Other Hardware","depth":3},{"id":"limitations","text":"Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""},{"id":"reparameterization/unireplknet","relation":"referenced","topic":""}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"Giant Convolutional Kernels","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"en","path":"/content/papers/en/reparameterization/replknet.271ac18a9d7c.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"a-guide-to-using-large-kernels-in-convolution","text":"A Guide to Using Large Kernels in Convolution","depth":2},{"id":"deep-large-kernels-remain-efficient","text":"Deep Large Kernels Remain Efficient","depth":3},{"id":"large-kernels-heavily-rely-on-residual-connections","text":"Large Kernels Heavily Rely on Residual Connections","depth":3},{"id":"large-kernels-require-reparameterization","text":"Large Kernels Require Reparameterization","depth":3},{"id":"large-kernels-significantly-enhance-downstream-task-performance","text":"Large Kernels Significantly Enhance Downstream Task Performance","depth":3},{"id":"large-kernels-are-effective-even-with-small-feature-maps","text":"Large Kernels Are Effective Even with Small Feature Maps","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"evaluation-of-large-kernels","text":"Evaluation of Large Kernels","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"large-kernel-cnns-have-larger-erf-than-deep-small-kernel-models","text":"Large Kernel CNNs Have Larger ERF Than Deep Small Kernel Models","depth":3},{"id":"shape-bias","text":"Shape Bias","depth":3},{"id":"limitations","text":"Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]},{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"Kernel Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"en","path":"/content/papers/en/classic-cnns/slak.5a1c38b2740d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"beyond-31","text":"Beyond 31","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"step-1-split-it-into-two-strips","text":"Step 1: Split It into Two Strips","depth":3},{"id":"step-2-a-little-sparser-a-little-wider","text":"Step 2: A Little Sparser, a Little Wider","depth":3},{"id":"what-slak-looks-like","text":"What SLaK Looks Like","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"running-it-for-real","text":"Running It for Real","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]},{"id":"reparameterization/unireplknet","category":"reparameterization","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"UniRepLKNet","subtitle":"Large Kernels Cross Boundaries","author":"Z. Yuan","paperDate":"2023-11-27","publishedOn":"2025-12-14","world":"external/unireplknet","paperTitle":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","paperUrl":"https://arxiv.org/abs/2311.15599","bibliography":{"title":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","authors":["Xiaohan Ding","Yiyuan Zhang","Yixiao Ge","Sijie Zhao","Lin Song","Xiangyu Yue","Ying Shan"],"url":"https://arxiv.org/abs/2311.15599","pdfUrl":"https://arxiv.org/pdf/2311.15599v2","codeUrl":"https://github.com/AILab-CVC/UniRepLKNet","preprint":{"id":"2311.15599","submittedOn":"2023-11-27","url":"https://arxiv.org/abs/2311.15599"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Ding_UniRepLKNet_A_Universal_Perception_Large-Kernel_ConvNet_for_Audio_Video_Point_CVPR_2024_paper.html"},"citation":"@misc{reparameterizationunireplknet2023,\n  title = {{UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition}},\n  author = {Xiaohan Ding and Yiyuan Zhang and Yixiao Ge and Sijie Zhao and Lin Song and Xiangyu Yue and Ying Shan},\n  year = {2023},\n  eprint = {2311.15599},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2311.15599}\n}\n","citationPath":"/content/papers/citations/reparameterization/unireplknet.df5ca5b9a0be.bib"},"thumbnail":{"src":"/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png","width":1680,"height":384},"locale":"en","path":"/content/papers/en/reparameterization/unireplknet.136ac87079c1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"one-33-layer-does-three-jobs","text":"One 3×3 Layer Does Three Jobs","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"baseline-architecture","text":"Baseline Architecture","depth":3},{"id":"guideline-1-deepen-with-cheap-structures","text":"Guideline 1: Deepen with Cheap Structures","depth":3},{"id":"guideline-2-reparameterize-with-small-dilated-convolutions","text":"Guideline 2: Reparameterize with Small Dilated Convolutions","depth":3},{"id":"guideline-3-choose-kernel-size-for-the-downstream-task","text":"Guideline 3: Choose Kernel Size for the Downstream Task","depth":3},{"id":"guideline-4-add-small-kernels-when-adding-depth","text":"Guideline 4: Add Small Kernels When Adding Depth","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"images","text":"Images","depth":3},{"id":"other-modalities","text":"Other Modalities","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""},{"id":"lightweight/senet","relation":"referenced","topic":""}]}],"text":"Convolution kernels keep growing. ConvNeXt uses 7×7, RepLKNet reaches 31×31, SLaK reaches 51×51, and UniRepLKNet settles back at 13×13. Accuracy keeps improving, while speed remains a problem.\n\nDepthwise convolution has few FLOPs, yet it slows rapidly as kernels grow. The authors measure a 71% to 78% throughput drop when standard depthwise convolution expands from 5×5 to 13×13, and 92% to 94% at 27×27. Results are similar across batch sizes and feature-map sizes.\n\nRepLKNet and UniRepLKNet address this with a custom CUDA operator, called LKA (Large Kernel Acceleration) in this paper. It helps on small feature maps but becomes slower on large ones. At 224×224 and batch size 128, LKA with 7×7 is 80% slower than unaccelerated 5×5, and with 13×13 it is 89% slower. Detection, segmentation, and super-resolution all need large feature maps.\n\nThis paper comes from Hefei University of Technology, Lingyang Industrial Internet, and Kochi University of Technology in Japan. It takes a different direction: compute convolution with matrix multiplication.\n\n\n> [!TIP]\n> \n> For earlier stops along the large-kernel route, see our notes:\n> \n> - **[[22.03] RepLKNet: Giant Convolutional Kernels](/blog/papers/reparameterization/replknet)**\n> - **[[22.07] SLaK: Kernel Pro Max](/blog/papers/classic-cnns/slak)**\n> - **[[23.11] UniRepLKNet: Large Kernels Cross Boundaries](/blog/papers/reparameterization/unireplknet)**\n\n\n## Defining the Problem\n\n### Convolution Waits for Data\n\n[![Figure 2 from the paper, with four panels. (a) A 5×5 depthwise convolution gathers values for a red output y from five rows of five cells, with thin red lines converging on y; each output collects 25 values from five noncontiguous rows. (b) WBMM's 4×4 windows divide an 8×8 feature map into four nonoverlapping colored regions W0 through W3; each window reads 16 values from one contiguous block. (c) Depthwise memory access has red boxes in rows 1 through 5 labeled fetch #1 through fetch #5, with notes for five independent reads, stride equal to width, a working set spanning multiple cache lines, and O(1) operations per byte: memory-bound. (d) WBMM takes four values from each of four rows in blue boxes, then window_partition and reshape arrange them into 16 contiguous cells requiring only fetch #1, with contiguous addresses, an L1-cache-sized window, and O(w²) operations per byte: compute-bound.](/content/papers/assets/a830da4662436ed9b9ed1985cd4a866d805da44ae503db57f3af5ed78d1f3f63.png)](/content/papers/assets/a830da4662436ed9b9ed1985cd4a866d805da44ae503db57f3af5ed78d1f3f63.png)\n\n*Figure 2 from Song et al., taken from [**PDF page 2**](https://arxiv.org/pdf/2607.02097v2#page=2). The original caption is omitted.*\n\nMemory access is the issue. Feature maps are stored row by row. Each depthwise-convolution output gathers $k$ values from each of $k$ surrounding rows. Those values are not contiguous in memory: the $k$ values within a row are adjacent, but reaching the next row requires a stride equal to the full row width.\n\nLarger kernels visit more rows and put more pressure on caches. Each value read supports just one multiply-add. GPU arithmetic capacity stays idle while data arrives.\n\n## Solving the Problem\n\n### Write Convolution as Matrix Multiplication\n\n[![Figure 1 from the paper: (a) depthwise convolution on the left shows input X as a large grid, with an orange 7×7 region around red cell (i, j), convolved with a 7×7 kernel W. An arrow says to gather 7² = 49 neighbors to produce y[i, j]. (b) WBMM on the right divides X into four colored 7×7 windows. A light-blue relative-position table R above has size C × (2·7 − 1)² = C × 169; an index arrow forms matrix M of size C × 49 × 49. A green Batched MatMul arrow multiplies input windows by M, producing output Y with the same window partition.](/content/papers/assets/5c20ec217f297d24ad8ce71ca79936d0599f7b460167e3bb89f52515ff1347ee.png)](/content/papers/assets/5c20ec217f297d24ad8ce71ca79936d0599f7b460167e3bb89f52515ff1347ee.png)\n\n*Figure 1 from Song et al., taken from [**PDF page 1**](https://arxiv.org/pdf/2607.02097v2#page=1). The original caption is omitted.*\n\nWBMM (Windowed Batch Matrix Multiplication) has three steps:\n\n1. **Partition windows**: split the feature map into nonoverlapping $w\\times w$ windows, such as 7×7. Flatten each into 49 values arranged contiguously in memory.\n2. **Look up weights**: each channel has a relative-position table $R$ of size $(2w-1)^2$. A 7×7 window uses 13×13 = 169 parameters. The weight used by window position $i$ for position $j$ depends only on their relative position; look it up to form a 49×49 matrix $M$.\n3. **Batched matrix multiplication**: all windows within a channel share $M$. Use channels as the batch dimension, and one `bmm` computes every window.\n\nWeights depend only on relative position, just as in convolution. The paper's Theorem 3.2 proves that on a bounded feature map, depthwise convolution can be written exactly as matrix multiplication with relative-position weights. Applied within each window, WBMM is equivalent to a 13×13 depthwise convolution with everything outside the window set to 0. Implement it and compare against a 13×13 convolution run window by window:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\nC, H, W, w = 4, 14, 14, 7                       # 4 channels, 14×14 feature map, 7×7 windows\nd = w * w                                        # 49 positions per window\nR = torch.randn(C, (2 * w - 1) ** 2)             # Relative-position table: 13×13 = 169 parameters per channel\n\n# Relative-position index: which table entry connects window position i to j\npos = torch.stack(torch.meshgrid(torch.arange(w), torch.arange(w), indexing=\"ij\")).flatten(1)  # 2 × 49\nrel = pos[:, None, :] - pos[:, :, None] + (w - 1)                                                # 2 × 49 × 49\nindex = rel[0] * (2 * w - 1) + rel[1]\n\nM = R[:, index]                                  # C × 49 × 49: form the weight matrix by lookup\n\n# Split the feature map into nonoverlapping windows, arranged as (C, window count, 49)\nx = torch.randn(1, C, H, W)\nnh, nw = H // w, W // w\nwindows = x.view(C, nh, w, nw, w).permute(0, 1, 3, 2, 4).reshape(C, nh * nw, d)\ny = torch.bmm(windows, M.transpose(1, 2))         # One batched matrix multiplication computes every window per channel\ny = y.view(C, nh, nw, w, w).permute(0, 1, 3, 2, 4).reshape(1, C, H, W)\n\n# Reference: 13×13 depthwise convolution, one window at a time, with zeros outside each window\nkernel = R.view(C, 1, 2 * w - 1, 2 * w - 1)\nref = torch.zeros_like(x)\nfor i in range(nh):\n    for j in range(nw):\n        patch = x[..., i * w:(i + 1) * w, j * w:(j + 1) * w]\n        ref[..., i * w:(i + 1) * w, j * w:(j + 1) * w] = F.conv2d(patch, kernel, padding=w - 1, groups=C)\n\nprint(\"Weight matrix:\", tuple(M.shape), \", formed by lookup from\", R.shape[1], \"parameters\")\nprint(\"Maximum difference from within-window 13×13 depthwise convolution:\", f\"{(y - ref).abs().max():.1e}\")\n```\n\n```text\nWeight matrix: (4, 49, 49) , formed by lookup from 169 parameters\nMaximum difference from within-window 13×13 depthwise convolution: 2.9e-06\n```\n\nThe results agree up to floating-point error. What differs is memory behavior: convolution gathers scattered neighbors one by one; WBMM reads a contiguous window and hands it to the GPU's highly optimized matrix multiplication.\n\n### Bigger Windows Run Faster\n\nThis brings a counterintuitive property. Larger depthwise kernels run slower, but larger WBMM windows run faster.\n\nA $w\\times w$ window reads $w^2$ values and performs $w^4$ multiply-adds, giving $w^2$ operations per value read. Larger windows raise the computation-to-read ratio and keep the GPU busier. The weight matrix $M$ has $w^4$ entries, but every window in the batch shares it, amortizing the read.\n\nWith 14×14 windows, each output takes 196 multiply-adds, nearly 8 times a 5×5 depthwise convolution, and sees 7.8 times the area. The operator benchmark uses an A800, FP32, and 256 channels. Comparing WBMM with 14×14 windows against standard 5×5 depthwise convolution:\n\n| Batch size | 14×14 feature map | 28×28 | 56×56 | 112×112 | 224×224 |\n| ---: | ---: | ---: | ---: | ---: | ---: |\n| 4 | 0.45× | 0.37× | 1.41× | 1.64× | 2.02× |\n| 16 | 0.62× | 1.56× | 1.80× | 2.00× | 2.07× |\n| 64 | 1.86× | 1.85× | 2.01× | 2.01× | 2.03× |\n| 128 | 3.17× | 2.00× | 2.06× | 1.97× | 2.01× |\n| 256 | 3.72× | 2.02× | 2.02× | 1.95× | — |\n\n*Table: from the paper's Table 1; values are WBMM speed relative to 5×5 depthwise convolution.*\n\nWith sufficiently large feature maps and batches, WBMM is about twice as fast. Small batches and small maps can be slower: at batch size 4 and 28×28, it runs at just 0.37×.\n\nThis differs greatly from window attention. Swin computes an input-dependent matrix and softmax for each sample and window. WBMM's $M$ is input-independent and shared across the batch; at inference, it can be precomputed and stored.\n\n### Let Windows Exchange Information\n\nNonoverlapping windows trap information inside each window. The authors interleave standard 3×3 depthwise convolutions between WBMM blocks to exchange information across neighboring windows.\n\nFor multiscale tasks such as segmentation, they also use hierarchical window reparameterization. During training, one large 14×14 window and four small 7×7 windows each have relative-position tables. Before inference, add the small-window matrices into diagonal blocks of the large-window matrix, producing one matrix at no extra inference cost. The idea parallels UniRepLKNet's merging of small kernels into a large one.\n\nThe full model starts from UniRepLKNet, replaces large kernels with WBMM, and adjusts the arrangement of WBMM and 3×3 blocks within each stage.\n\n## Discussion\n\n### ImageNet\n\n| Model | Training time per epoch | Training memory | top-1 |\n| --- | ---: | ---: | ---: |\n| WBMM-T | **6:18** | **15.0 GB** | 83.2% |\n| UniRepLKNet-T (with LKA) | 9:03 | 16.6 GB | 83.2% |\n| UniRepLKNet-T (without LKA) | 11:03 | 16.6 GB | 83.2% |\n| WBMM-S | **8:35** | **20.4 GB** | 83.9% |\n| UniRepLKNet-S (with LKA) | 12:10 | 22.3 GB | 83.9% |\n\n*Table: selected rows from the paper's Table 5. Eight A800s.*\n\nAccuracy is identical, with training more than 1.4 times as fast. Without LKA, the gap is larger. Across four model sizes, training speedups range from 1.31 to 1.88 times—the numbers in the abstract.\n\nThis compares against the entire UniRepLKNet, though, including several small dilated reparameterization branches during training. An appendix table makes a fairer comparison, keeping everything fixed except the spatial operator:\n\n| Spatial operator | Time per epoch | top-1 | ADE20K mIoU | Inference (images/s) |\n| --- | ---: | ---: | ---: | ---: |\n| WBMM (7×7 windows) | **6:20** | 83.0% | 48.3 | **1833** |\n| WBMM (14×14 hierarchical) | 6:31 | 83.2% | **48.8** | 1842 |\n| 13×13 depthwise + LKA | 6:43 | 83.1% | 48.3 | 1662 |\n| 13×13 depthwise (without LKA) | 9:11 | 83.1% | 48.3 | 1305 |\n| 13×13 depthwise + reparameterization + LKA | 10:21 | 83.2% | 48.7 | 1662 |\n| Original SLaK | 22:03 | 83.3% | 47.4 | 549 |\n| 7×7 window attention | 8:38 | 83.3% | 48.1 | 1245 |\n\n*Table: selected rows from the paper's Table 8, all Tiny-sized with the same architecture.*\n\nAgainst plain 13×13 depthwise convolution with LKA, WBMM gives only 6% faster training and 10% faster inference. Much of the previous table's 1.4× gain comes from removing UniRepLKNet's multibranch reparameterization: 10:21 becomes 6:20. Against depthwise convolution without LKA, WBMM is more than 1.4 times as fast.\n\nWBMM's real value is reaching or exceeding LKA speed without writing a custom CUDA operator.\n\n### Detection and Segmentation\n\n| Method | ADE20K mIoU (single-scale) | COCO box AP | COCO mask AP |\n| --- | ---: | ---: | ---: |\n| UniRepLKNet-T | 48.6 | 51.8 | 44.9 |\n| WBMM-T (7×7) | 48.3 | 51.6 | 44.8 |\n| WBMM-T (hierarchical) | **48.8** | **51.9** | **45.1** |\n\n*Table: from the paper's Tables 6 and 7. ADE20K uses UperNet; COCO uses Cascade Mask R-CNN.*\n\nThe plain 7×7-window variant trails UniRepLKNet by 0.1 to 0.3. Hierarchical reparameterization brings it 0.1 to 0.2 ahead, also adding 3M to 4M parameters. Detection and segmentation performance is roughly tied; WBMM's gain is speed.\n\n### Other Hardware\n\nThe authors also measure other devices. On an A800, WBMM runs 1.01 to 1.28 times as fast as UniRepLKNet with LKA. On an Intel i7-13700K CPU, where LKA is unavailable, BLAS gives WBMM a 1.03 to 1.48× speedup. On Jetson Orin Nano, the GPU gets 1.19 to 3.12× in FP16, and the CPU 1.33 to 2.44× in INT8. Every hardware platform optimizes matrix multiplication, making this especially useful on edge devices.\n\n### Limitations\n\nThe appendix lists several limitations. The weight matrix $M$ grows with the fourth power of window side length. At 14×14, a 256-channel layer takes about 37.5 MB; beyond 28×28 it no longer fits. Window size also depends on the task: plain 14×14 performs worse than plain 7×7 on ADE20K. Input-independent weights may be insufficient for tasks needing content adaptation.\n\n## Conclusion\n\nWBMM preserves what the model sees. Within each window, it is a 13×13 depthwise convolution; the change is how data is arranged in memory and which operator computes it. Batched matrix multiplication turns time spent waiting for data into time spent computing, so larger windows can run faster.\n\nRead the tables with the baseline in mind. The abstract's 1.31 to 1.88× compares against the full UniRepLKNet. Changing only the operator gives 6% faster training than depthwise convolution with LKA. The real saving is the custom CUDA implementation.\n\nIf high-resolution speed is holding back your large-kernel model, try this approach. PyTorch's `bmm` is enough. The first half of the demonstration contains the full core logic, with no CUDA extension to compile.\n","imageSizes":{"/content/papers/assets/a830da4662436ed9b9ed1985cd4a866d805da44ae503db57f3af5ed78d1f3f63.png":{"width":1032,"height":864},"/content/papers/assets/5c20ec217f297d24ad8ce71ca79936d0599f7b460167e3bb89f52515ff1347ee.png":{"width":1017,"height":360}}}