{"notes":[{"id":"reparameterization/unireplknet","category":"reparameterization","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"UniRepLKNet","subtitle":"大核跨界","author":"Z. Yuan","paperDate":"2023-11-27","publishedOn":"2025-12-14","world":"external/unireplknet","paperTitle":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","paperUrl":"https://arxiv.org/abs/2311.15599","bibliography":{"title":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","authors":["Xiaohan Ding","Yiyuan Zhang","Yixiao Ge","Sijie Zhao","Lin Song","Xiangyu Yue","Ying Shan"],"url":"https://arxiv.org/abs/2311.15599","pdfUrl":"https://arxiv.org/pdf/2311.15599v2","codeUrl":"https://github.com/AILab-CVC/UniRepLKNet","preprint":{"id":"2311.15599","submittedOn":"2023-11-27","url":"https://arxiv.org/abs/2311.15599"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Ding_UniRepLKNet_A_Universal_Perception_Large-Kernel_ConvNet_for_Audio_Video_Point_CVPR_2024_paper.html"},"citation":"@misc{reparameterizationunireplknet2023,\n  title = {{UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition}},\n  author = {Xiaohan Ding and Yiyuan Zhang and Yixiao Ge and Sijie Zhao and Lin Song and Xiangyu Yue and Ying Shan},\n  year = {2023},\n  eprint = {2311.15599},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2311.15599}\n}\n","citationPath":"/content/papers/citations/reparameterization/unireplknet.df5ca5b9a0be.bib"},"thumbnail":{"src":"/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png","width":1680,"height":384},"locale":"zh-hans","path":"/content/papers/zh-hans/reparameterization/unireplknet.6c5d5c7f5107.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"一层-33-做了三件事","text":"一层 3×3 做了三件事","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"基准架构","text":"基准架构","depth":3},{"id":"准则一加深用便宜的结构","text":"准则一：加深用便宜的结构","depth":3},{"id":"准则二用空洞小卷积重参数化","text":"准则二：用空洞小卷积重参数化","depth":3},{"id":"准则三卷积核大小看下游任务","text":"准则三：卷积核大小看下游任务","depth":3},{"id":"准则四加深时加的是小卷积核","text":"准则四：加深时，加的是小卷积核","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"图像","text":"图像","depth":3},{"id":"其他模态","text":"其他模态","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""},{"id":"lightweight/senet","relation":"referenced","topic":""}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"巨无霸卷积核","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"zh-hans","path":"/content/papers/zh-hans/reparameterization/replknet.63f1e846afbb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"大核卷积使用说明书","text":"大核卷积使用说明书","depth":2},{"id":"深度大核卷积仍然高效","text":"深度大核卷积仍然高效","depth":3},{"id":"大核卷积十分依赖残差连接","text":"大核卷积十分依赖残差连接","depth":3},{"id":"大核卷积需要重参数化","text":"大核卷积需要重参数化","depth":3},{"id":"大核卷积明显提升下游任务性能","text":"大核卷积明显提升下游任务性能","depth":3},{"id":"大核卷积对于小特征图的效果依然很好","text":"大核卷积对于小特征图的效果依然很好","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"网络架构","text":"网络架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"大核卷积评估","text":"大核卷积评估","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"大核-cnn-比深度小核模型有更大的-erf","text":"大核 CNN 比深度小核模型有更大的 ERF","depth":3},{"id":"形状偏差","text":"形状偏差","depth":3},{"id":"限制","text":"限制","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"大卷积核与简化骨架"}]},{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"卷积核 Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/slak.d84cfaf376c9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"31-之后","text":"31 之后","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"第一步拆成两根长条","text":"第一步：拆成两根长条","depth":3},{"id":"第二步稀疏一点宽一点","text":"第二步：稀疏一点，宽一点","depth":3},{"id":"slak-的样子","text":"SLaK 的样子","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任务","text":"下游任务","depth":3},{"id":"实际跑起来","text":"实际跑起来","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]},{"id":"lightweight/senet","category":"lightweight","legacyPath":"lightweight/1709-senet","legacySlug":"1709-senet","legacyPaths":[],"title":"SENet","subtitle":"挤！用力挤！","author":"Z. Yuan","paperDate":"2017-09-05","publishedOn":"2024-06-28","paperTitle":"Squeeze-and-Excitation Networks","paperUrl":"https://arxiv.org/abs/1709.01507","bibliography":{"title":"Squeeze-and-Excitation Networks","authors":["Jie Hu","Li Shen","Samuel Albanie","Gang Sun","Enhua Wu"],"url":"https://arxiv.org/abs/1709.01507","pdfUrl":"https://arxiv.org/pdf/1709.01507","codeUrl":null,"preprint":{"id":"1709.01507","submittedOn":"2017-09-05","url":"https://arxiv.org/abs/1709.01507"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1709.01507"},"citation":"@misc{lightweightsenet2017,\n  title = {{Squeeze-and-Excitation Networks}},\n  author = {Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},\n  year = {2017},\n  eprint = {1709.01507},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1709.01507}\n}\n","citationPath":"/content/papers/citations/lightweight/senet.1766a7978dde.bib"},"thumbnail":{"src":"/content/papers/assets/322f16ed8c782ffbdd7d7db3e8292c991908e3543dc69e1addd9238e22484363.jpg","width":1688,"height":394},"locale":"zh-hans","path":"/content/papers/zh-hans/lightweight/senet.fa3515c1688a.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型架构","text":"模型架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"影像分类效果展示","text":"影像分类效果展示","depth":3},{"id":"轻量模型效果展示","text":"轻量模型效果展示","depth":3},{"id":"squeeze-的重要性","text":"Squeeze 的重要性","depth":3},{"id":"excitation-的重要性","text":"Excitation 的重要性","depth":3},{"id":"其他实验结果","text":"其他实验结果","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v3","relation":"comparison","topic":"轻量网络里的通道注意力"},{"id":"lightweight/pp-lcnet","relation":"comparison","topic":"轻量网络里的通道注意力"}]}],"text":"RepLKNet 把卷积核放大到 31×31 之后，大卷积核成了热门题目。SLaK 拆成长条做到 51×51，也有人把大卷积核搬到检测、分割等各种任务上。\n\n这篇的第一作者 Xiaohan Ding 是 RepLKNet 和 RepVGG 的作者，这次他在腾讯 AI Lab，和香港中文大学的研究者一起回头检查大卷积核网络的架构。他们发现，现有的大卷积核网络，架构大多直接沿用别人的设计，RepLKNet 跟着 Swin Transformer，SLaK 跟着 ConvNeXt，而 ConvNeXt 本身用的是 7×7 的中型卷积核。专门为大卷积核设计的架构，还很少有人研究。\n\n论文还想回答另一个问题。Transformer 已经在图像、语音、点云、时间序列这些模态都站稳了，卷积网络出了图像，还有没有竞争力？\n\n\n> [!TIP]\n> \n> 大卷积核这条路的前两站：\n> \n> - **[[22.03] RepLKNet：巨无霸卷积核](/blog/papers/reparameterization/replknet)**\n> - **[[22.07] SLaK：卷积核 Pro Max](/blog/papers/classic-cnns/slak)**\n\n\n## 定义问题\n\n### 一层 3×3 做了三件事\n\n作者从小卷积核网络的老问题开始想。在一个由 3×3 卷积堆起来的网络里多加一层，同时会发生三件事：\n\n1. 感受野变大\n2. 空间特征的抽象层次变高，例如从角点、纹理变成物体的形状\n3. 网络变深，多了参数和非线性，表达能力变强\n\n小卷积核网络只能靠叠层数来扩大感受野，所以这三件事绑在一起。大卷积核不一样，它不用变深就看得很广。几层大卷积核就足以撑出够大的有效感受野，剩下的算力可以留给更有效率的结构，去处理第二件和第三件事。\n\n作者认为大卷积核网络应该把这三件事拆开来设计，接下来的四条准则都从这里出发。\n\n## 解决问题\n\n### 基准架构\n\n[![原论文 Figure 1：左边是四个阶段的主干，Stage 1 到 Stage 4 之间以 2 倍降采样相接，输入先经过 4 倍降采样；中间展开 Stage 3，LarK Block（橘色）和 SmaK Block（蓝色）交错排列，每个 LarK Block 后面接两个 SmaK Block；右边虚线框是 LarK Block 的内部，依序是 Dilated Reparam Block 构成的 depthwise 卷积、BN、包含平均池化、ReLU 和 sigmoid 相乘的 SE Block、FFN、BN，最后经过 drop path 和残差相加。](/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png)](/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png)\n\n*原论文 Figure 1（Ding 等人），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/2311.15599v2#page=2)。原图注未收录。*\n\n先看作者拿来做实验的基准。四个阶段，通道数是 $C$、$2C$、$4C$、$8C$，区块数 $(3, 3, 9, 3)$，和 ConvNeXt-T 一样。区块长得像 ConvNeXt，一个 depthwise 卷积接一个带 GRN 的 FFN，只是把 LayerNorm 换成 BatchNorm，推理时可以并入卷积。第一个阶段用 3×3，后三个阶段用 13×13。\n\n做架构探索时，ImageNet 只训练 100 个 epoch，再拿去 ADE20K 上接 UperNet 做分割。作者特别强调要看分割，因为大卷积核的好处在下游任务上比较明显，只看 ImageNet 准确率容易看不出差别。\n\n### 准则一：加深用便宜的结构\n\n既然感受野交给大卷积核，网络要变深，就用不需要大范围空间运算的结构。作者在每个区块里多加一个小结构，试了四种：\n\n| 加的结构 | 吞吐量（张／秒） | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 不加 | 1954 | 81.2% | 45.1 |\n| 瓶颈结构（1×1、DW 3×3、1×1） | 1716 | 81.5% | 46.3 |\n| 两个 1×1 | 1745 | 81.3% | 46.2 |\n| 两个 DW 3×3 | 1659 | 81.3% | 45.4 |\n| SE Block | **1863** | **81.6%** | **46.5** |\n\n*表：摘自原论文 Table 1。A100，批量 128。*\n\nSE Block 同时做到了通道之间的交流和空间上的聚合，它的空间聚合就是全局平均池化，非常便宜，结果最好也最快。\n\n\n> [!TIP]\n> \n> SE Block 怎么用全局平均池化替通道加权，可以看这篇：\n> \n> - **[[17.09] SENet：挤！用力挤！](/blog/papers/lightweight/senet)**\n\n\n### 准则二：用空洞小卷积重参数化\n\nRepLKNet 已经发现，大卷积核训练时旁边要并联一个小卷积核，帮忙抓小尺度的样式；训练完把 BN 并入卷积，小卷积核就能补零加进大卷积核，推理时只剩一层。\n\n这篇多想了一步。除了小尺度的样式，还有一种稀疏的样式，特征图上的某个像素，可能和几格外的像素比较有关，和紧邻的反而没那么有关。抓这种样式正好是空洞卷积（dilated convolution）的专长，间隔 $r$ 的空洞卷积，每个采样点之间隔了 $r-1$ 个像素。\n\n[![原论文 Figure 2：上半部是结构的角度，输入同时送进五个并联分支，9×9 间隔 1、5×5 间隔 1、5×5 间隔 2、3×3 间隔 3、3×3 间隔 4，各自接一个 BN 后相加；右边箭头写着 Re-parameterize，指向单个 9×9 间隔 1 的卷积。下半部是参数的角度，每个分支的权重画成方格，空洞卷积的权重摊开成中间夹着空格的稀疏方格，全部叠加起来变成一个 9×9 的方格，颜色深浅代表不同分支叠上去的权重。](/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png)](/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png)\n\n*原论文 Figure 2（Ding 等人），取自 [**PDF 第 4 页**](https://arxiv.org/pdf/2311.15599v2#page=4)。原图注未收录。*\n\n图中用 $K=9$ 当例子，实际的网络用 $K=13$。重点在下半部，间隔 $r$ 的 $k\\times k$ 卷积，等同于一个 $(k-1)r+1$ 大小、中间插满 0 的稀疏卷积核。既然它也是一个卷积核，就能和大卷积核相加。\n\n先验证第一步，空洞卷积和摊开后的稀疏卷积核算出来一模一样：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ndef dilate(w, r):\n    \"\"\"把 k×k 的权重摊开：相邻两个权重之间插 r-1 个 0。\"\"\"\n    c, _, k, _ = w.shape\n    size = (k - 1) * r + 1\n    big = torch.zeros(c, 1, size, size)\n    big[:, :, ::r, ::r] = w\n    return big\n\ntorch.manual_seed(0)\nc = 4\nx = torch.randn(1, c, 32, 32)\nw = torch.randn(c, 1, 3, 3)\ny_dilated = F.conv2d(x, w, padding=3, dilation=3, groups=c)   # 3×3、间隔 3 的空洞卷积\ny_sparse = F.conv2d(x, dilate(w, 3), padding=3, groups=c)     # 摊开成 7×7 的稀疏卷积核\nprint(\"摊开后的卷积核：\", tuple(dilate(w, 3).shape[-2:]), \"其中非零的格子\", int((dilate(w, 3)[0] != 0).sum()))\nprint(\"两种算法的最大差距：\", f\"{(y_dilated - y_sparse).abs().max():.1e}\")\n```\n\n```text\n摊开后的卷积核： (7, 7) 其中非零的格子 9\n两种算法的最大差距： 0.0e+00\n```\n\n再照论文的默认值，13×13 主卷积核旁边并联五条分支，卷积核大小 $k=(5,7,3,3,3)$，间隔 $r=(1,2,3,4,5)$。训练时各算各的再相加；推理前把每条分支摊开、补零到 13×13，全部加进主卷积核：\n\n```python\nK = 13\nbranches = [(5, 1), (7, 2), (3, 3), (3, 4), (3, 5)]          # (卷积核大小 k, 间隔 r)\nw_main = torch.randn(c, 1, K, K)\nw_small = [torch.randn(c, 1, k, k) for k, _ in branches]\n\n# 训练时：一个 13×13 加五条空洞小卷积，各自算完再相加\ny_train = F.conv2d(x, w_main, padding=K // 2, groups=c)\nfor (k, r), w in zip(branches, w_small):\n    y_train += F.conv2d(x, w, padding=r * (k - 1) // 2, dilation=r, groups=c)\n\n# 推理前：每条分支摊开、补零到 13×13，全部加进主卷积核\nmerged = w_main.clone()\nfor (k, r), w in zip(branches, w_small):\n    big = dilate(w, r)\n    pad = (K - big.shape[-1]) // 2\n    merged += F.pad(big, [pad] * 4)\n    print(f\"{k}×{k}、间隔 {r} → 等效 {big.shape[-1]}×{big.shape[-1]}\")\ny_infer = F.conv2d(x, merged, padding=K // 2, groups=c)\nprint(\"输出数值的大小：\", f\"{y_train.abs().max():.1f}\")\nprint(\"合并成一个 13×13 之后的最大差距：\", f\"{(y_train - y_infer).abs().max():.1e}\")\n```\n\n```text\n5×5、间隔 1 → 等效 5×5\n7×7、间隔 2 → 等效 13×13\n3×3、间隔 3 → 等效 7×7\n3×3、间隔 4 → 等效 9×9\n3×3、间隔 5 → 等效 11×11\n输出数值的大小： 59.9\n合并成一个 13×13 之后的最大差距： 2.7e-05\n```\n\n六个卷积合成一个，差距只剩浮点数加法顺序造成的舍入误差。实际的模块每条分支后面还有 BN，推理前先把 BN 的缩放和平移并入卷积的权重和偏置，做法和 RepVGG 一样。\n\n\n> [!TIP]\n> \n> BN 怎么并入卷积、并联分支怎么合成一条，可以看这篇：\n> \n> - **[[21.01] RepVGG：让 VGG 再次伟大](/blog/papers/reparameterization/repvgg)**\n\n\n这样做有没有用？作者拿两种没有空洞的版本来比，分支数一样，一个用一样的卷积核大小 $(5,7,3,3,3)$，一个用一样的等效大小 $(5,13,7,9,11)$。推理时三者都是同一个 13×13，差别只在训练：\n\n| 重参数化方式 | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: |\n| 不做 | 81.44 ± 0.04% | 45.78 ± 0.05 |\n| 空洞小卷积 | **81.63 ± 0.02%** | **46.37 ± 0.10** |\n| 一样的卷积核大小，不空洞 | 81.55 ± 0.01% | 46.07 ± 0.07 |\n| 一样的等效大小，不空洞 | 81.59 ± 0.02% | 46.17 ± 0.04 |\n\n*表：摘自原论文 Table 2，三次训练的平均与标准差。*\n\n空洞版本最好，mIoU 比不做重参数化多了 0.6。作者的解读是，帮上忙的是空洞卷积抓稀疏样式的能力，光是多几个小卷积核，或是多几种感受野大小，效果都比较差。\n\n### 准则三：卷积核大小看下游任务\n\n接着看每个阶段该用多大的卷积核：\n\n| 各阶段卷积核 | 吞吐量（张／秒） | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 3、13、13、13（基准） | 1863 | 81.6% | **46.5** |\n| 3、11、11、11 | 1876 | 81.6% | 45.5 |\n| 3、15、15、15 | 1851 | 81.7% | 45.9 |\n| 3、3、13、13 | 2006 | 81.7% | 46.1 |\n| 13、13、13、13 | 1547 | 81.6% | 44.9 |\n\n*表：摘自原论文 Table 3 的部分行。*\n\nImageNet 准确率几乎没动，都在 81.6% 上下；mIoU 却从 44.9 跳到 46.5。只看分类的话，会以为卷积核大小没什么影响。\n\n奇怪的是，卷积核再放大到 15，或第一个阶段也用 13，mIoU 反而下降。作者怀疑问题出在 UperNet。它会把骨干前几层的特征和最后几层组合起来，默认前几层只带局部信息；前几层用了大卷积核，特征不再那么局部，组合起来就没那么有用。\n\n作者验证的方法是只让 UperNet 用第四个阶段的输出，单独评估最后的特征质量。这时 15 的 mIoU 最高，是 42.7；第一阶段也用 13 的版本是 42.4，和基准一样；11 最低，是 41.9。大卷积核没有让特征变差，只是让前几层的特征不合 UperNet 的胃口。\n\n所以卷积核要多大，要看下游接的是什么。在常见的图像任务里，作者默认中、高层用 13×13。\n\n### 准则四：加深时，加的是小卷积核\n\n最后是放大模型。作者把第三阶段的区块数从 9 加到 27，问题是多出来的 18 个区块要不要也用大卷积核。这里把用 Dilated Reparam Block 的区块叫 LarK Block，用 DW 3×3 的叫 SmaK Block：\n\n| 第三阶段的组成 | 吞吐量（张／秒） | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 9 个 LarK（原本的浅版本） | 1863 | 81.6% | 46.5 |\n| 27 个 LarK | 1145 | 82.3% | **49.0** |\n| 14 个 LarK、13 个 SmaK 交错 | 1229 | 82.3% | 48.8 |\n| 9 个 LarK、18 个 SmaK | 1264 | 82.3% | 48.8 |\n| 同上，SmaK 拿掉 DW 3×3 | 1289 | 82.2% | 47.8 |\n\n*表：摘自原论文 Table 4。*\n\n全部用大卷积核，mIoU 只多了 0.2，速度却慢了将近一成。9 个大卷积核已经够撑感受野，加深的部分用小卷积核就好。\n\n最后一行很有意思。SmaK 里的 3×3 拿掉，速度只快一点，mIoU 却掉了一个百分点。这些小卷积核对感受野几乎没贡献，可是它们让空间特征的抽象层次往上叠，正好呼应开头说的第二件事。\n\n最终的 UniRepLKNet 就是照这个方式排的。从 S 开始，第三阶段是 9 个 LarK 加 18 个 SmaK，每个 LarK 后面接两个 SmaK，也就是架构图中间的样子；最小的 T 是 9 加 9。\n\n## 讨论\n\n### 图像\n\n先看只用 ImageNet-1K 训练的结果，吞吐量在 A100 上用批量 128 量：\n\n| 模型 | 参数量 | 吞吐量（张／秒） | top-1 |\n| --- | ---: | ---: | ---: |\n| SLaK-T | 30M | 1312 | 82.5% |\n| InternImage-T | 30M | 1292 | 83.5% |\n| UniRepLKNet-T | 31M | **1804** | 83.2% |\n| ConvNeXt-S | 50M | 1182 | 83.1% |\n| UniRepLKNet-S | 56M | 1265 | **83.9%** |\n\n*表：摘自原论文 Table 6 的部分行。*\n\nUniRepLKNet-T 比 InternImage-T 低 0.3 个百分点，速度却快了四成。用 ImageNet-22K 预训练、384 分辨率的最大版本，UniRepLKNet-XL 做到 88.0%，和 InternImage-XL 一样，吞吐量 131 对 114。\n\n下游任务也不错。ADE20K 上 UniRepLKNet-T 的 mIoU 是 48.6（单尺度），ConvNeXt-T 是 46.0；XL 做到 55.6（多尺度）。COCO 上 XL 的 box AP 是 56.4。\n\n### 其他模态\n\n要把同一个网络用在其他数据上，作者的做法很直接，把各种数据都整理成 $B\\times C'\\times H\\times W$ 的张量，只改第一层的输入通道数 $C'$，其他部分完全不动。\n\n- **语音**：$T$ 个时间格、$F$ 个频率格的频谱，直接当成单通道的 $T\\times F$ 图像。\n- **时间序列**：先用 Corrformer 的嵌入层投影到潜在空间，再整形成单通道的二维图。\n- **点云**：用几层卷积产生 X、Y、Z 三个视角的投影图，当成 224×224 的三通道图像。\n- **视频**：把 16 张 224×224 的帧排成 4×4 的拼图，变成一张 896×896 的大图。\n\n视频那一项最暴力。帧与帧之间的时间关系，全靠卷积核在拼图上跨过边界去看。\n\n四种模态的结果放在一起看：\n\n| 任务 | UniRepLKNet-S | 对照 |\n| --- | --- | --- |\n| 语音指令识别（Speech Commands V2） | 98.5%，55.5M，没有预训练 | Audio-MAE 98.3%，86.2M，AudioSet-2M 预训练 |\n| 全球气温预测（MSE，越低越好） | 7.602 | Corrformer 7.709 |\n| 3D 形状分类（ModelNet-40 OA） | 93.2% | DGCNN、KPConv 92.9% |\n| 动作识别（Kinetics-400） | 54.8%，没有预训练 | MViTv2-B 81.2%，ImageNet-1K 预训练 |\n\n*表：摘自原论文 Table 9 到 Table 12 的部分行。*\n\n语音、时间序列和点云都打平或小赢专门设计的模型。视频差得很远，作者也承认落后专门的视频模型；他们拿来比的是同样号称通用的 ImageBind（50.0%）和 Meta-Transformer（47.3%），这两个还用了大量预训练数据。\n\n卷积核大小在这里也很重要。同样的预处理和训练设置，换成其他卷积网络：\n\n| 模型（卷积核） | 语音 | 视频 |\n| --- | ---: | ---: |\n| ResNet-101（3） | 73.6% | 41.3% |\n| ConvNeXt-S（7） | 94.3% | 48.5% |\n| UniRepLKNet-S（11） | 94.7% | 51.7% |\n| UniRepLKNet-S（13） | **98.5%** | **54.8%** |\n\n*表：摘自原论文 Table 13 的部分列。*\n\nResNet-101 在语音上只有 73.6%，卷积核放大到 13 就到了 98.5%。频谱和拼图都需要一次看得很广，这正是大卷积核的长处。\n\n### 代价\n\n作者在结论写明了一个限制，空洞分支在训练时要花更多资源。五条分支各自做卷积、各自过 BN，推理时能合成一条，训练时省不掉。\n\n推理的速度也要看实现。大卷积核的 depthwise 卷积需要官方提供的加速实现，GitHub 上有用户[反馈训练很慢](https://github.com/AILab-CVC/UniRepLKNet/issues/15)，作者回复的第一个问题就是有没有装那个实现。也有人[在 4090 上量](https://github.com/AILab-CVC/UniRepLKNet/issues/22)，发现 UniRepLKNet-S 并没有比 ConvNeXt-B 快，作者回答论文是在 A100 上量的，速度排名和硬件有关。\n\n## 结论\n\nUniRepLKNet 的四条准则，其实都在讲同一件事。大卷积核负责看得广，小卷积核负责叠出层次，SE 这类便宜的结构负责加深，各做各的，不要让一种运算同时扛三件事。\n\nDilated Reparam Block 是最容易带走的部分。训练时多挂几条空洞小卷积，推理时合成一个大卷积核，部署端完全看不出差别。上面那段程序就是全部的合并逻辑，再补上 BN 的融合就完整了。\n\n另一头的多模态实验更出人意料。把语音频谱当成图片、把视频切成拼图，什么专用设计都不加，卷积网络照样能在好几个 Transformer 的主场打平，甚至小赢。\n","imageSizes":{"/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png":{"width":960,"height":621},"/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png":{"width":1680,"height":384}}}