{"notes":[{"id":"reparameterization/unireplknet","category":"reparameterization","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"UniRepLKNet","subtitle":"大核跨界","author":"Z. Yuan","paperDate":"2023-11-27","publishedOn":"2025-12-14","world":"external/unireplknet","paperTitle":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","paperUrl":"https://arxiv.org/abs/2311.15599","bibliography":{"title":"UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition","authors":["Xiaohan Ding","Yiyuan Zhang","Yixiao Ge","Sijie Zhao","Lin Song","Xiangyu Yue","Ying Shan"],"url":"https://arxiv.org/abs/2311.15599","pdfUrl":"https://arxiv.org/pdf/2311.15599v2","codeUrl":"https://github.com/AILab-CVC/UniRepLKNet","preprint":{"id":"2311.15599","submittedOn":"2023-11-27","url":"https://arxiv.org/abs/2311.15599"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Ding_UniRepLKNet_A_Universal_Perception_Large-Kernel_ConvNet_for_Audio_Video_Point_CVPR_2024_paper.html"},"citation":"@misc{reparameterizationunireplknet2023,\n  title = {{UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition}},\n  author = {Xiaohan Ding and Yiyuan Zhang and Yixiao Ge and Sijie Zhao and Lin Song and Xiangyu Yue and Ying Shan},\n  year = {2023},\n  eprint = {2311.15599},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2311.15599}\n}\n","citationPath":"/content/papers/citations/reparameterization/unireplknet.df5ca5b9a0be.bib"},"thumbnail":{"src":"/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png","width":1680,"height":384},"locale":"zh-hant","path":"/content/papers/zh-hant/reparameterization/unireplknet.e887d30226f9.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"一層-33-做了三件事","text":"一層 3×3 做了三件事","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"基準架構","text":"基準架構","depth":3},{"id":"準則一加深用便宜的結構","text":"準則一：加深用便宜的結構","depth":3},{"id":"準則二用空洞小卷積重參數化","text":"準則二：用空洞小卷積重參數化","depth":3},{"id":"準則三卷積核大小看下游任務","text":"準則三：卷積核大小看下游任務","depth":3},{"id":"準則四加深時加的是小卷積核","text":"準則四：加深時，加的是小卷積核","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"影像","text":"影像","depth":3},{"id":"其他模態","text":"其他模態","depth":3},{"id":"代價","text":"代價","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""},{"id":"lightweight/senet","relation":"referenced","topic":""}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"巨無霸卷積核","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"zh-hant","path":"/content/papers/zh-hant/reparameterization/replknet.16fd94d59060.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"大核卷積使用說明書","text":"大核卷積使用說明書","depth":2},{"id":"深度大核卷積仍然高效","text":"深度大核卷積仍然高效","depth":3},{"id":"大核卷積十分依賴殘差連接","text":"大核卷積十分依賴殘差連接","depth":3},{"id":"大核卷積需要重新參數化","text":"大核卷積需要重新參數化","depth":3},{"id":"大核卷積明顯提升下游任務性能","text":"大核卷積明顯提升下游任務性能","depth":3},{"id":"大核卷積對於小特徵圖的效果依然很好","text":"大核卷積對於小特徵圖的效果依然很好","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"網路架構","text":"網路架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"大核卷積評估","text":"大核卷積評估","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"大核-cnn-比深度小核模型有更大的-erf","text":"大核 CNN 比深度小核模型有更大的 ERF","depth":3},{"id":"形狀偏差","text":"形狀偏差","depth":3},{"id":"限制","text":"限制","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"大卷積核與簡化骨架"}]},{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"卷積核 Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/slak.9c5ac51ae21b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"31-之後","text":"31 之後","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"第一步拆成兩根長條","text":"第一步：拆成兩根長條","depth":3},{"id":"第二步稀疏一點寬一點","text":"第二步：稀疏一點，寬一點","depth":3},{"id":"slak-的樣子","text":"SLaK 的樣子","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任務","text":"下游任務","depth":3},{"id":"實際跑起來","text":"實際跑起來","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]},{"id":"lightweight/senet","category":"lightweight","legacyPath":"lightweight/1709-senet","legacySlug":"1709-senet","legacyPaths":[],"title":"SENet","subtitle":"擠！用力擠！","author":"Z. Yuan","paperDate":"2017-09-05","publishedOn":"2024-06-28","paperTitle":"Squeeze-and-Excitation Networks","paperUrl":"https://arxiv.org/abs/1709.01507","bibliography":{"title":"Squeeze-and-Excitation Networks","authors":["Jie Hu","Li Shen","Samuel Albanie","Gang Sun","Enhua Wu"],"url":"https://arxiv.org/abs/1709.01507","pdfUrl":"https://arxiv.org/pdf/1709.01507","codeUrl":null,"preprint":{"id":"1709.01507","submittedOn":"2017-09-05","url":"https://arxiv.org/abs/1709.01507"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1709.01507"},"citation":"@misc{lightweightsenet2017,\n  title = {{Squeeze-and-Excitation Networks}},\n  author = {Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},\n  year = {2017},\n  eprint = {1709.01507},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1709.01507}\n}\n","citationPath":"/content/papers/citations/lightweight/senet.1766a7978dde.bib"},"thumbnail":{"src":"/content/papers/assets/322f16ed8c782ffbdd7d7db3e8292c991908e3543dc69e1addd9238e22484363.jpg","width":1688,"height":394},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/senet.dd5fabe0868c.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"影像分類效果展示","text":"影像分類效果展示","depth":3},{"id":"輕量模型效果展示","text":"輕量模型效果展示","depth":3},{"id":"squeeze-的重要性","text":"Squeeze 的重要性","depth":3},{"id":"excitation-的重要性","text":"Excitation 的重要性","depth":3},{"id":"其他實驗結果","text":"其他實驗結果","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v3","relation":"comparison","topic":"輕量網路裡的通道注意力"},{"id":"lightweight/pp-lcnet","relation":"comparison","topic":"輕量網路裡的通道注意力"}]}],"text":"RepLKNet 把卷積核放大到 31×31 之後，大卷積核成了熱門題目。SLaK 拆成長條做到 51×51，也有人把大卷積核搬到偵測、分割等各種任務上。\n\n這篇的第一作者 Xiaohan Ding 是 RepLKNet 和 RepVGG 的作者，這次他在騰訊 AI Lab，和香港中文大學的研究者一起回頭檢查大卷積核網路的架構。他們發現，現有的大卷積核網路，架構大多直接沿用別人的設計，RepLKNet 跟著 Swin Transformer，SLaK 跟著 ConvNeXt，而 ConvNeXt 本身用的是 7×7 的中型卷積核。專門為大卷積核設計的架構，還很少有人研究。\n\n論文還想回答另一個問題。Transformer 已經在影像、語音、點雲、時間序列這些模態都站穩了，卷積網路出了影像，還有沒有競爭力？\n\n\n> [!TIP]\n> \n> 大卷積核這條路的前兩站：\n> \n> - **[[22.03] RepLKNet：巨無霸卷積核](/blog/papers/reparameterization/replknet)**\n> - **[[22.07] SLaK：卷積核 Pro Max](/blog/papers/classic-cnns/slak)**\n\n\n## 定義問題\n\n### 一層 3×3 做了三件事\n\n作者從小卷積核網路的老問題開始想。在一個由 3×3 卷積堆起來的網路裡多加一層，同時會發生三件事：\n\n1. 感受野變大\n2. 空間特徵的抽象層次變高，例如從角落、紋理變成物體的形狀\n3. 網路變深，多了參數和非線性，表達能力變強\n\n小卷積核網路只能靠疊層數來擴大感受野，所以這三件事綁在一起。大卷積核不一樣，它不用變深就看得很廣。幾層大卷積核就足以撐出夠大的有效感受野，剩下的算力可以留給更有效率的結構，去處理第二件和第三件事。\n\n作者認為大卷積核網路應該把這三件事拆開來設計，接下來的四條準則都從這裡出發。\n\n## 解決問題\n\n### 基準架構\n\n[![原論文 Figure 1：左邊是四個階段的主幹，Stage 1 到 Stage 4 之間以 2 倍降採樣相接，輸入先經過 4 倍降採樣；中間展開 Stage 3，LarK Block（橘色）和 SmaK Block（藍色）交錯排列，每個 LarK Block 後面接兩個 SmaK Block；右邊虛線框是 LarK Block 的內部，依序是 Dilated Reparam Block 構成的 depthwise 卷積、BN、包含平均池化、ReLU 和 sigmoid 相乘的 SE Block、FFN、BN，最後經過 drop path 和殘差相加。](/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png)](/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png)\n\n*原論文 Figure 1（Ding 等人），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/2311.15599v2#page=2)。圖說未收錄。*\n\n先看作者拿來做實驗的基準。四個階段，通道數是 $C$、$2C$、$4C$、$8C$，區塊數 $(3, 3, 9, 3)$，和 ConvNeXt-T 一樣。區塊長得像 ConvNeXt，一個 depthwise 卷積接一個帶 GRN 的 FFN，只是把 LayerNorm 換成 BatchNorm，推論時可以併進卷積。第一個階段用 3×3，後三個階段用 13×13。\n\n做架構探索時，ImageNet 只訓練 100 個 epoch，再拿去 ADE20K 上接 UperNet 做分割。作者特別強調要看分割，因為大卷積核的好處在下游任務上比較明顯，只看 ImageNet 準確率容易看不出差別。\n\n### 準則一：加深用便宜的結構\n\n既然感受野交給大卷積核，網路要變深，就用不需要大範圍空間運算的結構。作者在每個區塊裡多加一個小結構，試了四種：\n\n| 加的結構 | 吞吐量（張／秒） | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 不加 | 1954 | 81.2% | 45.1 |\n| 瓶頸結構（1×1、DW 3×3、1×1） | 1716 | 81.5% | 46.3 |\n| 兩個 1×1 | 1745 | 81.3% | 46.2 |\n| 兩個 DW 3×3 | 1659 | 81.3% | 45.4 |\n| SE Block | **1863** | **81.6%** | **46.5** |\n\n*表：摘自原論文 Table 1。A100，批次 128。*\n\nSE Block 同時做到了通道之間的交流和空間上的聚合，它的空間聚合就是全域平均池化，非常便宜，結果最好也最快。\n\n\n> [!TIP]\n> \n> SE Block 怎麼用全域平均池化替通道加權，可以看這篇：\n> \n> - **[[17.09] SENet：擠！用力擠！](/blog/papers/lightweight/senet)**\n\n\n### 準則二：用空洞小卷積重參數化\n\nRepLKNet 已經發現，大卷積核訓練時旁邊要並聯一個小卷積核，幫忙抓小尺度的樣式；訓練完把 BN 併進卷積，小卷積核就能補零加進大卷積核，推論時只剩一層。\n\n這篇多想了一步。除了小尺度的樣式，還有一種稀疏的樣式，特徵圖上的某個像素，可能和幾格外的像素比較有關，和緊鄰的反而沒那麼有關。抓這種樣式正好是空洞卷積（dilated convolution）的專長，間隔 $r$ 的空洞卷積，每個取樣點之間隔了 $r-1$ 個像素。\n\n[![原論文 Figure 2：上半部是結構的角度，輸入同時送進五個並聯分支，9×9 間隔 1、5×5 間隔 1、5×5 間隔 2、3×3 間隔 3、3×3 間隔 4，各自接一個 BN 後相加；右邊箭頭寫著 Re-parameterize，指向單一個 9×9 間隔 1 的卷積。下半部是參數的角度，每個分支的權重畫成方格，空洞卷積的權重攤開成中間夾著空格的稀疏方格，全部疊加起來變成一個 9×9 的方格，顏色深淺代表不同分支疊上去的權重。](/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png)](/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png)\n\n*原論文 Figure 2（Ding 等人），取自 [**PDF 第 4 頁**](https://arxiv.org/pdf/2311.15599v2#page=4)。圖說未收錄。*\n\n圖中用 $K=9$ 當例子，實際的網路用 $K=13$。重點在下半部，間隔 $r$ 的 $k\\times k$ 卷積，等同於一個 $(k-1)r+1$ 大小、中間插滿 0 的稀疏卷積核。既然它也是一個卷積核，就能和大卷積核相加。\n\n先驗證第一步，空洞卷積和攤開後的稀疏卷積核算出來一模一樣：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ndef dilate(w, r):\n    \"\"\"把 k×k 的權重攤開：相鄰兩個權重之間插 r-1 個 0。\"\"\"\n    c, _, k, _ = w.shape\n    size = (k - 1) * r + 1\n    big = torch.zeros(c, 1, size, size)\n    big[:, :, ::r, ::r] = w\n    return big\n\ntorch.manual_seed(0)\nc = 4\nx = torch.randn(1, c, 32, 32)\nw = torch.randn(c, 1, 3, 3)\ny_dilated = F.conv2d(x, w, padding=3, dilation=3, groups=c)   # 3×3、間隔 3 的空洞卷積\ny_sparse = F.conv2d(x, dilate(w, 3), padding=3, groups=c)     # 攤開成 7×7 的稀疏卷積核\nprint(\"攤開後的卷積核：\", tuple(dilate(w, 3).shape[-2:]), \"其中非零的格子\", int((dilate(w, 3)[0] != 0).sum()))\nprint(\"兩種算法的最大差距：\", f\"{(y_dilated - y_sparse).abs().max():.1e}\")\n```\n\n```text\n攤開後的卷積核： (7, 7) 其中非零的格子 9\n兩種算法的最大差距： 0.0e+00\n```\n\n再照論文的預設值，13×13 主卷積核旁邊並聯五條分支，卷積核大小 $k=(5,7,3,3,3)$，間隔 $r=(1,2,3,4,5)$。訓練時各算各的再相加；推論前把每條分支攤開、補零到 13×13，全部加進主卷積核：\n\n```python\nK = 13\nbranches = [(5, 1), (7, 2), (3, 3), (3, 4), (3, 5)]          # (卷積核大小 k, 間隔 r)\nw_main = torch.randn(c, 1, K, K)\nw_small = [torch.randn(c, 1, k, k) for k, _ in branches]\n\n# 訓練時：一個 13×13 加五條空洞小卷積，各自算完再相加\ny_train = F.conv2d(x, w_main, padding=K // 2, groups=c)\nfor (k, r), w in zip(branches, w_small):\n    y_train += F.conv2d(x, w, padding=r * (k - 1) // 2, dilation=r, groups=c)\n\n# 推論前：每條分支攤開、補零到 13×13，全部加進主卷積核\nmerged = w_main.clone()\nfor (k, r), w in zip(branches, w_small):\n    big = dilate(w, r)\n    pad = (K - big.shape[-1]) // 2\n    merged += F.pad(big, [pad] * 4)\n    print(f\"{k}×{k}、間隔 {r} → 等效 {big.shape[-1]}×{big.shape[-1]}\")\ny_infer = F.conv2d(x, merged, padding=K // 2, groups=c)\nprint(\"輸出數值的大小：\", f\"{y_train.abs().max():.1f}\")\nprint(\"合併成一個 13×13 之後的最大差距：\", f\"{(y_train - y_infer).abs().max():.1e}\")\n```\n\n```text\n5×5、間隔 1 → 等效 5×5\n7×7、間隔 2 → 等效 13×13\n3×3、間隔 3 → 等效 7×7\n3×3、間隔 4 → 等效 9×9\n3×3、間隔 5 → 等效 11×11\n輸出數值的大小： 59.9\n合併成一個 13×13 之後的最大差距： 2.7e-05\n```\n\n六個卷積合成一個，差距只剩浮點數加法順序造成的捨入誤差。實際的模組每條分支後面還有 BN，推論前先把 BN 的縮放和平移併進卷積的權重和偏差，做法和 RepVGG 一樣。\n\n\n> [!TIP]\n> \n> BN 怎麼併進卷積、並聯分支怎麼合成一條，可以看這篇：\n> \n> - **[[21.01] RepVGG：讓 VGG 再次偉大](/blog/papers/reparameterization/repvgg)**\n\n\n這樣做有沒有用？作者拿兩種沒有空洞的版本來比，分支數一樣，一個用一樣的卷積核大小 $(5,7,3,3,3)$，一個用一樣的等效大小 $(5,13,7,9,11)$。推論時三者都是同一個 13×13，差別只在訓練：\n\n| 重參數化方式 | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: |\n| 不做 | 81.44 ± 0.04% | 45.78 ± 0.05 |\n| 空洞小卷積 | **81.63 ± 0.02%** | **46.37 ± 0.10** |\n| 一樣的卷積核大小，不空洞 | 81.55 ± 0.01% | 46.07 ± 0.07 |\n| 一樣的等效大小，不空洞 | 81.59 ± 0.02% | 46.17 ± 0.04 |\n\n*表：摘自原論文 Table 2，三次訓練的平均與標準差。*\n\n空洞版本最好，mIoU 比不做重參數化多了 0.6。作者的解讀是，幫上忙的是空洞卷積抓稀疏樣式的能力，光是多幾個小卷積核，或是多幾種感受野大小，效果都比較差。\n\n### 準則三：卷積核大小看下游任務\n\n接著看每個階段該用多大的卷積核：\n\n| 各階段卷積核 | 吞吐量（張／秒） | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 3、13、13、13（基準） | 1863 | 81.6% | **46.5** |\n| 3、11、11、11 | 1876 | 81.6% | 45.5 |\n| 3、15、15、15 | 1851 | 81.7% | 45.9 |\n| 3、3、13、13 | 2006 | 81.7% | 46.1 |\n| 13、13、13、13 | 1547 | 81.6% | 44.9 |\n\n*表：摘自原論文 Table 3 的部分列。*\n\nImageNet 準確率幾乎沒動，都在 81.6% 上下；mIoU 卻從 44.9 跳到 46.5。只看分類的話，會以為卷積核大小沒什麼影響。\n\n奇怪的是，卷積核再放大到 15，或第一個階段也用 13，mIoU 反而下降。作者懷疑問題出在 UperNet。它會把骨幹前幾層的特徵和最後幾層組合起來，預設前幾層只帶局部資訊；前幾層用了大卷積核，特徵不再那麼局部，組合起來就沒那麼有用。\n\n作者驗證的方法是只讓 UperNet 用第四個階段的輸出，單獨評估最後的特徵品質。這時 15 的 mIoU 最高，是 42.7；第一階段也用 13 的版本是 42.4，和基準一樣；11 最低，是 41.9。大卷積核沒有讓特徵變差，只是讓前幾層的特徵不合 UperNet 的胃口。\n\n所以卷積核要多大，要看下游接的是什麼。在常見的影像任務裡，作者預設中、高層用 13×13。\n\n### 準則四：加深時，加的是小卷積核\n\n最後是放大模型。作者把第三階段的區塊數從 9 加到 27，問題是多出來的 18 個區塊要不要也用大卷積核。這裡把用 Dilated Reparam Block 的區塊叫 LarK Block，用 DW 3×3 的叫 SmaK Block：\n\n| 第三階段的組成 | 吞吐量（張／秒） | ImageNet | ADE20K mIoU |\n| --- | ---: | ---: | ---: |\n| 9 個 LarK（原本的淺版本） | 1863 | 81.6% | 46.5 |\n| 27 個 LarK | 1145 | 82.3% | **49.0** |\n| 14 個 LarK、13 個 SmaK 交錯 | 1229 | 82.3% | 48.8 |\n| 9 個 LarK、18 個 SmaK | 1264 | 82.3% | 48.8 |\n| 同上，SmaK 拿掉 DW 3×3 | 1289 | 82.2% | 47.8 |\n\n*表：摘自原論文 Table 4。*\n\n全部用大卷積核，mIoU 只多了 0.2，速度卻慢了將近一成。9 個大卷積核已經夠撐感受野，加深的部分用小卷積核就好。\n\n最後一列很有意思。SmaK 裡的 3×3 拿掉，速度只快一點，mIoU 卻掉了一個百分點。這些小卷積核對感受野幾乎沒貢獻，可是它們讓空間特徵的抽象層次往上疊，正好呼應開頭說的第二件事。\n\n最終的 UniRepLKNet 就是照這個方式排的。從 S 開始，第三階段是 9 個 LarK 加 18 個 SmaK，每個 LarK 後面接兩個 SmaK，也就是架構圖中間的樣子；最小的 T 是 9 加 9。\n\n## 討論\n\n### 影像\n\n先看只用 ImageNet-1K 訓練的結果，吞吐量在 A100 上用批次 128 量：\n\n| 模型 | 參數量 | 吞吐量（張／秒） | top-1 |\n| --- | ---: | ---: | ---: |\n| SLaK-T | 30M | 1312 | 82.5% |\n| InternImage-T | 30M | 1292 | 83.5% |\n| UniRepLKNet-T | 31M | **1804** | 83.2% |\n| ConvNeXt-S | 50M | 1182 | 83.1% |\n| UniRepLKNet-S | 56M | 1265 | **83.9%** |\n\n*表：摘自原論文 Table 6 的部分列。*\n\nUniRepLKNet-T 比 InternImage-T 低 0.3 個百分點，速度卻快了四成。用 ImageNet-22K 預訓練、384 解析度的最大版本，UniRepLKNet-XL 做到 88.0%，和 InternImage-XL 一樣，吞吐量 131 對 114。\n\n下游任務也不錯。ADE20K 上 UniRepLKNet-T 的 mIoU 是 48.6（單尺度），ConvNeXt-T 是 46.0；XL 做到 55.6（多尺度）。COCO 上 XL 的 box AP 是 56.4。\n\n### 其他模態\n\n要把同一個網路用在其他資料上，作者的做法很直接，把各種資料都整理成 $B\\times C'\\times H\\times W$ 的張量，只改第一層的輸入通道數 $C'$，其他部分完全不動。\n\n- **語音**：$T$ 個時間格、$F$ 個頻率格的頻譜，直接當成單通道的 $T\\times F$ 影像。\n- **時間序列**：先用 Corrformer 的嵌入層投影到潛在空間，再整形成單通道的二維圖。\n- **點雲**：用幾層卷積產生 X、Y、Z 三個視角的投影圖，當成 224×224 的三通道影像。\n- **影片**：把 16 張 224×224 的影格排成 4×4 的拼圖，變成一張 896×896 的大圖。\n\n影片那一項最暴力。幀與幀之間的時間關係，全靠卷積核在拼圖上跨過邊界去看。\n\n四種模態的結果放在一起看：\n\n| 任務 | UniRepLKNet-S | 對照 |\n| --- | --- | --- |\n| 語音指令辨識（Speech Commands V2） | 98.5%，55.5M，沒有預訓練 | Audio-MAE 98.3%，86.2M，AudioSet-2M 預訓練 |\n| 全球氣溫預測（MSE，越低越好） | 7.602 | Corrformer 7.709 |\n| 3D 形狀分類（ModelNet-40 OA） | 93.2% | DGCNN、KPConv 92.9% |\n| 動作辨識（Kinetics-400） | 54.8%，沒有預訓練 | MViTv2-B 81.2%，ImageNet-1K 預訓練 |\n\n*表：摘自原論文 Table 9 到 Table 12 的部分列。*\n\n語音、時間序列和點雲都打平或小贏專門設計的模型。影片差得很遠，作者也承認落後專門的影片模型；他們拿來比的是同樣號稱通用的 ImageBind（50.0%）和 Meta-Transformer（47.3%），這兩個還用了大量預訓練資料。\n\n卷積核大小在這裡也很重要。同樣的前處理和訓練設定，換成其他卷積網路：\n\n| 模型（卷積核） | 語音 | 影片 |\n| --- | ---: | ---: |\n| ResNet-101（3） | 73.6% | 41.3% |\n| ConvNeXt-S（7） | 94.3% | 48.5% |\n| UniRepLKNet-S（11） | 94.7% | 51.7% |\n| UniRepLKNet-S（13） | **98.5%** | **54.8%** |\n\n*表：摘自原論文 Table 13 的部分欄位。*\n\nResNet-101 在語音上只有 73.6%，卷積核放大到 13 就到了 98.5%。頻譜和拼圖都需要一次看得很廣，這正是大卷積核的長處。\n\n### 代價\n\n作者在結論寫明了一個限制，空洞分支在訓練時要花更多資源。五條分支各自做卷積、各自過 BN，推論時能合成一條，訓練時省不掉。\n\n推論的速度也要看實作。大卷積核的 depthwise 卷積需要官方提供的加速實作，GitHub 上有使用者[回報訓練很慢](https://github.com/AILab-CVC/UniRepLKNet/issues/15)，作者回覆的第一個問題就是有沒有裝那個實作。也有人[在 4090 上量](https://github.com/AILab-CVC/UniRepLKNet/issues/22)，發現 UniRepLKNet-S 並沒有比 ConvNeXt-B 快，作者回答論文是在 A100 上量的，速度排名和硬體有關。\n\n## 結論\n\nUniRepLKNet 的四條準則，其實都在講同一件事。大卷積核負責看得廣，小卷積核負責疊出層次，SE 這類便宜的結構負責加深，各做各的，不要讓一種運算同時扛三件事。\n\nDilated Reparam Block 是最容易帶走的部分。訓練時多掛幾條空洞小卷積，推論時合成一個大卷積核，部署端完全看不出差別。上面那段程式就是全部的合併邏輯，再補上 BN 的融合就完整了。\n\n另一頭的多模態實驗更出人意料。把語音頻譜當成圖片、把影片切成拼圖，什麼專用設計都不加，卷積網路照樣能在好幾個 Transformer 的主場打平，甚至小贏。\n","imageSizes":{"/content/papers/assets/95a64d551639000b4d79a5746c5aa0f05d914227d468929959577bade902ad80.png":{"width":960,"height":621},"/content/papers/assets/8dc0f2f7d7d64f92bea8485e2e00428225297bb13404dee50fe50f28385d34c8.png":{"width":1680,"height":384}}}