{"notes":[{"id":"classic-cnns/overlock","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"OverLoCK","subtitle":"先看大局","author":"Z. Yuan","paperDate":"2025-02-27","publishedOn":"2026-02-24","paperTitle":"OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels","paperUrl":"https://arxiv.org/abs/2502.20087","bibliography":{"title":"OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels","authors":["Meng Lou","Yizhou Yu"],"url":"https://arxiv.org/abs/2502.20087","pdfUrl":"https://arxiv.org/pdf/2502.20087v3","codeUrl":"https://github.com/LMMMEng/OverLoCK","preprint":{"id":"2502.20087","submittedOn":"2025-02-27","url":"https://arxiv.org/abs/2502.20087"},"publication":{"venue":"CVPR","year":2025,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2025/html/Lou_OverLoCK_An_Overview-first-Look-Closely-next_ConvNet_with_Context-Mixing_Dynamic_Kernels_CVPR_2025_paper.html"},"citation":"@misc{classiccnnsoverlock2025,\n  title = {{OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels}},\n  author = {Meng Lou and Yizhou Yu},\n  year = {2025},\n  eprint = {2502.20087},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.20087}\n}\n","citationPath":"/content/papers/citations/classic-cnns/overlock.a1eecba8875b.bib"},"thumbnail":{"src":"/content/papers/assets/99458c439c8f5458671d09ec6d68f52f31a20702b6a0bdba99e2fe33dbf86ea7.png","width":2112,"height":681},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/overlock.c982182e0497.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"中间层缺一个方向","text":"中间层缺一个方向","depth":3},{"id":"卷积核看不远","text":"卷积核看不远","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"拆成三个子网络","text":"拆成三个子网络","depth":3},{"id":"contmix把全图塞进卷积核","text":"ContMix：把全图塞进卷积核","depth":3},{"id":"四个尺寸","text":"四个尺寸","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"检测与分割","text":"检测与分割","depth":3},{"id":"一步一步加上去","text":"一步一步加上去","depth":3},{"id":"和其他动态算子比","text":"和其他动态算子比","depth":3},{"id":"实际用起来","text":"实际用起来","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/internimage","relation":"referenced","topic":""},{"id":"classic-cnns/dcnv4","relation":"referenced","topic":""}]},{"id":"classic-cnns/internimage","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InternImage","subtitle":"大力出奇迹？","author":"Z. Yuan","paperDate":"2022-11-10","publishedOn":"2025-09-04","world":"external/internimage","paperTitle":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","paperUrl":"https://arxiv.org/abs/2211.05778","bibliography":{"title":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","authors":["Wenhai Wang","Jifeng Dai","Zhe Chen","Zhenhang Huang","Zhiqi Li","Xizhou Zhu","Xiaowei Hu","Tong Lu","Lewei Lu","Hongsheng Li","Xiaogang Wang","Yu Qiao"],"url":"https://arxiv.org/abs/2211.05778","pdfUrl":"https://arxiv.org/pdf/2211.05778v4","codeUrl":null,"preprint":{"id":"2211.05778","submittedOn":"2022-11-10","url":"https://arxiv.org/abs/2211.05778"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Wang_InternImage_Exploring_Large-Scale_Vision_Foundation_Models_With_Deformable_Convolutions_CVPR_2023_paper.html"},"citation":"@misc{classiccnnsinternimage2022,\n  title = {{InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions}},\n  author = {Wenhai Wang and Jifeng Dai and Zhe Chen and Zhenhang Huang and Zhiqi Li and Xizhou Zhu and Xiaowei Hu and Tong Lu and Lewei Lu and Hongsheng Li and Xiaogang Wang and Yu Qiao},\n  year = {2022},\n  eprint = {2211.05778},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2211.05778}\n}\n","citationPath":"/content/papers/citations/classic-cnns/internimage.4baca0899bc9.bib"},"thumbnail":{"src":"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png","width":1012,"height":1179},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/internimage.f5dfee837064.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"卷积输在哪里","text":"卷积输在哪里","depth":3},{"id":"dcnv2-为什么不能直接用","text":"DCNv2 为什么不能直接用","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"dcnv3-的三个修改","text":"DCNv3 的三个修改","depth":3},{"id":"叠成网络","text":"叠成网络","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"检测与分割","text":"检测与分割","depth":3},{"id":"消融softmax-那一项","text":"消融：softmax 那一项","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/deformable-convnets","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""}]},{"id":"classic-cnns/dcnv4","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"DCNv4","subtitle":"权重不设上限","author":"Z. Yuan","paperDate":"2024-01-11","publishedOn":"2025-12-29","paperTitle":"Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications","paperUrl":"https://arxiv.org/abs/2401.06197","bibliography":{"title":"Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications","authors":["Yuwen Xiong","Zhiqi Li","Yuntao Chen","Feng Wang","Xizhou Zhu","Jiapeng Luo","Wenhai Wang","Tong Lu","Hongsheng Li","Yu Qiao","Lewei Lu","Jie Zhou","Jifeng Dai"],"url":"https://arxiv.org/abs/2401.06197","pdfUrl":"https://arxiv.org/pdf/2401.06197v1","codeUrl":"https://github.com/OpenGVLab/DCNv4","preprint":{"id":"2401.06197","submittedOn":"2024-01-11","url":"https://arxiv.org/abs/2401.06197"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Xiong_Efficient_Deformable_ConvNets_Rethinking_Dynamic_and_Sparse_Operator_for_Vision_CVPR_2024_paper.html"},"citation":"@misc{classiccnnsdcnv42024,\n  title = {{Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications}},\n  author = {Yuwen Xiong and Zhiqi Li and Yuntao Chen and Feng Wang and Xizhou Zhu and Jiapeng Luo and Wenhai Wang and Tong Lu and Hongsheng Li and Yu Qiao and Lewei Lu and Jie Zhou and Jifeng Dai},\n  year = {2024},\n  eprint = {2401.06197},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2401.06197}\n}\n","citationPath":"/content/papers/citations/classic-cnns/dcnv4.f17f87f8ee11.bib"},"thumbnail":{"src":"/content/papers/assets/fe768f9a1b9df098fd7de1ed2c34ba0713cb83936e024d45ca26e85a2b340ff1.png","width":1020,"height":522},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/dcnv4.6cd84adfd6b4.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"只采样-9-个点为什么不快","text":"只采样 9 个点，为什么不快","depth":3},{"id":"softmax-是借来的","text":"softmax 是借来的","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"一拿掉-softmax","text":"一、拿掉 softmax","depth":3},{"id":"二让-gpu-少搬数据","text":"二、让 GPU 少搬数据","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"算子本身","text":"算子本身","depth":3},{"id":"flashinternimage","text":"FlashInternImage","depth":3},{"id":"换进别的网络","text":"换进别的网络","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/internimage","relation":"referenced","topic":""},{"id":"classic-cnns/deformable-convnets","relation":"referenced","topic":""}]}],"text":"人看一张照片，通常不是从左上角一格一格看下去。我们会先扫一眼，知道这是一只水豚泡在温泉里，再回头去看它头上那颗橘子。认知科学把这叫做由上而下的注意力（top-down attention），先有粗略的全貌，再用它引导细看。\n\n卷积网络的骨干不是这样运作的。金字塔式的架构一层层降采样，每一层只能用前面几层的结果，中间层在处理特征的时候，还不知道这张图整体在讲什么。\n\n这篇来自香港大学，作者是 Meng Lou 和 Yizhou Yu。他们想做一个纯卷积的骨干，让网络也能「先看大局，再看细节」。\n\n## 定义问题\n\n### 中间层缺一个方向\n\n作者用 Grad-CAM 看了 Swin-T、ConvNeXt-T 和 VMamba-T 的中间层，发现离分类器比较远的第三阶段，常常抓不到正确类别的物体。这些网络都能捕捉长距离关系，只是中间层缺少一个来自高层语义的指引。\n\n以前也有人在视觉模型里加由上而下的信号，主流做法是循环，把最后的输出接回前面再跑一次。这样就要多算一轮，在骨干网络上划不来。\n\n### 卷积核看不远\n\n第二个问题在算子。有了全图的语义，还要有办法把它用进卷积核里。\n\n大卷积核和一般的动态卷积，看的范围都被卷积核大小限制住了，输入分辨率变大，能看到的比例就跟着变小。可变形卷积能看得比较远，可是采样点一旦散开，卷积原本的局部归纳偏置也就没了。作者要的算子，权重要随输入变化，要能用上全图的信息，还要保有局部的归纳偏置。\n\n## 解决问题\n\n### 拆成三个子网络\n\n[![原论文 Figure 3：下方是 Base-Net，输入一张头上顶着橘子的水豚照片，依序经过三组 Embedding 和 Basic Block，分辨率从 H/4 降到 H/16。上方左边是 Overview-Net，接在 Base-Net 之后，一个 Embedding 加几个 Basic Block，直接降到 H/32，输出一个绿色立方体标示为 Context Prior。上方右边是 Focus-Net，接收 Base-Net 的输出，经过 Dynamic Block、Embedding、Dynamic Block，从 H/16 到 H/32；红色箭头代表 context guidance flow，把 context prior 送进 Focus-Net 的每个 Dynamic Block。](/content/papers/assets/d32df0756ccbd0c769259f897b5ad9cf4c8079758897be9a2147cd65a4d46ce1.png)](/content/papers/assets/d32df0756ccbd0c769259f897b5ad9cf4c8079758897be9a2147cd65a4d46ce1.png)\n\n*原论文 Figure 3（Lou 与 Yu），取自 [**PDF 第 4 页**](https://arxiv.org/pdf/2502.20087v3#page=4)，CC BY 4.0。原图注未收录。*\n\n作者把网络拆成三个子网络，叫做深层分解策略（Deep-stage Decomposition Strategy，DDS）：\n\n- **Base-Net**：一般的前三个阶段，把图像降到 $H/16\\times W/16$，产生中层特征。\n- **Overview-Net**：很轻的一段，拿 Base-Net 的输出直接降到 $H/32$，快速得到一份粗糙但有语义的全图理解，叫做 context prior。这就是「先看大局」。\n- **Focus-Net**：比较深的一段，同样从 Base-Net 的输出出发，一边往下做，一边用 context prior 指引每个区块。这就是「再看细节」。\n\n所以这里其实藏了两个骨干，Base-Net 接 Overview-Net 是一个，Base-Net 接 Focus-Net 是另一个，两者共用 Base-Net。预训练时两个子网络各接一个分类头，都算分类损失；用在下游任务时，Overview-Net 不再需要额外监督，特征金字塔取 Base-Net 的 $H/4$、$H/8$，加上 Focus-Net 的 $H/16$、$H/32$。\n\ncontext prior 在 Focus-Net 里会一路更新。每个区块把它和特征图接在一起处理，出来时再拆开，更新后的 context prior 交给下一个区块。为了避免越传越淡，每次都会加回最初那份，写成 $P_{i+1}=\\alpha P'_i+\\beta P_0$，$\\alpha$ 和 $\\beta$ 是可学的标量。\n\nBase-Net 和 Overview-Net 用的区块里，有一个来自 UniRepLKNet 的 Dilated RepConv，也就是用空洞小卷积重参数化的大卷积核。\n\n### ContMix：把全图塞进卷积核\n\nFocus-Net 的核心是一个新的动态卷积，叫 ContMix（Context-Mixing Dynamic Convolution）。\n\n[![原论文 Figure 5：(a) ContMix 的流程图。输入特征 X 经过 1×1 卷积和重塑得到 Q；另一路先做自适应平均池化，再经过 1×1 卷积得到 K。Q 转置后和 K 做矩阵乘法，得到大小 G×HW×S² 的亲和力矩阵 A，其中一行用黄色虚线框起来；这一行经过线性层 W_d 和 softmax，变成 G×1×K² 的向量，再重塑成 K×K 的「全局感知卷积核」，和输入做逐元素相乘再加总；下方还有一条 K×K RepConv 的旁路，最后相加输出。(b) 一张泡在温泉里的水豚照片，切成四块，每块有一个彩色星号代表区域中心；中间一个 3×3 的网格，九个蓝色圆点代表「全局感知权重」，红色虚线从四个区域中心连到网格中央。](/content/papers/assets/99458c439c8f5458671d09ec6d68f52f31a20702b6a0bdba99e2fe33dbf86ea7.png)](/content/papers/assets/99458c439c8f5458671d09ec6d68f52f31a20702b6a0bdba99e2fe33dbf86ea7.png)\n\n*原论文 Figure 5（Lou 与 Yu），取自 [**PDF 第 5 页**](https://arxiv.org/pdf/2502.20087v3#page=5)，CC BY 4.0。原图注未收录。*\n\n跟着图 (a) 走一次：\n\n1. 输入 $X$ 经过 1×1 卷积得到 $Q$，每个位置一个查询向量，共 $HW$ 个。\n2. $X$ 另外用自适应平均池化压成 $S\\times S$ 个区域中心，再经过 1×1 卷积得到 $K$。论文设 $S=7$，也就是 49 个区域中心。\n3. $Q$ 和 $K$ 相乘，得到亲和力矩阵 $A$，大小是 $HW\\times S^2$。第 $i$ 行记录了第 $i$ 个位置和 49 个区域中心各有多像。\n4. 每一行经过一个线性层 $W_d$，从 49 个数变成 $K^2$ 个数，再做 softmax，就是这个位置专属的 $K\\times K$ 卷积核。\n5. 用这些卷积核对输入做卷积，每个位置用自己的那一个。\n\n通道会分成 $G$ 组，每组算自己的亲和力和卷积核，同一组的通道共用。放进 OverLoCK 的动态区块时，$Q$ 来自特征图，$K$ 则来自 context prior，Overview-Net 看到的全貌就是从这里进到卷积核里。\n\n重点在第 4 步。卷积核的每个权重，都是由「这个位置和全图 49 个区域中心的关系」算出来的，所以权重里带着全图的信息。可是卷积本身还是在 $K\\times K$ 的邻域里做，局部的归纳偏置没有丢掉。图 (b) 画的就是这件事，3×3 的格子还在原地，决定格子权重的却是四个角落的区域中心。\n\n我们写一个最简化的版本验证。改动左上角一个像素，看右下角的输出会不会变：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\nC, H, W = 8, 16, 16\nS, K, G = 4, 3, 2                    # 4×4 个区域中心、3×3 的动态卷积核、通道分 2 组\nWq = torch.nn.Conv2d(C, C, 1)\nWk = torch.nn.Conv2d(C, C, 1)\nWd = torch.nn.Linear(S * S, K * K)   # 把 16 个相似度变成 9 个卷积核权重\n\ndef contmix(x):\n    q = Wq(x[None])[0].flatten(1)                                   # C × HW：每个位置的查询\n    k = Wk(F.adaptive_avg_pool2d(x[None], S))[0].flatten(1)         # C × S²：区域中心\n    out = []\n    for qg, kg, xg in zip(q.chunk(G), k.chunk(G), x.chunk(G)):\n        A = qg.T @ kg                                               # HW × S²：每个位置和每个区域中心的相似度\n        D = Wd(A).softmax(-1)                                       # HW × K²：每个位置自己的 3×3 卷积核\n        patches = F.unfold(xg[None], K, padding=K // 2)[0]          # 每个位置周围的 3×3 邻居\n        patches = patches.view(C // G, K * K, H * W)\n        out.append((patches * D.T[None]).sum(1).view(C // G, H, W))\n    return torch.cat(out)\n\nstatic = torch.nn.Conv2d(C, C, K, padding=K // 2, groups=C)       # 一般的 3×3 depthwise 卷积\n\nx = torch.randn(C, H, W)\nx2 = x.clone()\nx2[:, 0, 0] += 5.0                                                  # 只改左上角那一个像素\n\nwith torch.no_grad():\n    for name, f in [(\"3×3 depthwise 卷积\", lambda t: static(t[None])[0]), (\"ContMix（3×3）\", contmix)]:\n        diff = (f(x2) - f(x))[:, 12, 12].abs().max().item()       # 看右下角 (12, 12) 的输出变了多少\n        print(f\"{name}：右下角输出的变化 {diff:.4f}\")\n```\n\n```text\n3×3 depthwise 卷积：右下角输出的变化 0.0000\nContMix（3×3）：右下角输出的变化 0.0274\n```\n\n同样是 3×3，一般卷积完全看不到 12 格外的变化，ContMix 看得到。左上角的像素改变了第一个区域中心，所有位置的亲和力跟着变，右下角那个位置的卷积核也就换了。\n\n实际的模块比这个复杂一些。官方代码还有一个 value 投影和相对位置偏置；一半的组用大卷积核，另一半用 5×5 的小卷积核；旁边再并联一条 Dilated RepConv。有装邻域注意力库 NATTEN 的话，应用动态卷积核那一步直接借用它的 CUDA 算子，计算方式和邻域注意力的加权加总一模一样。差别在权重怎么来，邻域注意力是查询和邻居的 key 相乘，ContMix 则是查询和全图的区域中心相乘。\n\n### 四个尺寸\n\n四个版本 XT、T、S、B，T 有 33M 参数、5.5 GFLOPs，B 有 95M、16.7 GFLOPs。Base-Net 的三个阶段用 17、15、13 的卷积核，Overview-Net 用 7，Focus-Net 用 13 和 7。\n\n## 讨论\n\n### ImageNet\n\nImageNet-1K 训练 300 个 epoch，配方照 DeiT。吞吐量用一张 L40S、批量 128、224×224 输入量：\n\n| 模型 | 参数量 | FLOPs | 吞吐量（张／秒） | top-1 |\n| --- | ---: | ---: | ---: | ---: |\n| ConvNeXt-T | 29M | 4.5G | 1507 | 82.1% |\n| UniRepLKNet-T | 31M | 4.9G | 1094 | 83.2% |\n| InternImage-T | 30M | 5.0G | 1084 | 83.5% |\n| MogaNet-S | 25M | 5.0G | 766 | 83.4% |\n| OverLoCK-T | 33M | 5.5G | 810 | **84.2%** |\n| ConvNeXt-S | 50M | 8.7G | 926 | 83.1% |\n| ConvNeXt-B | 89M | 15.4G | 608 | 83.8% |\n| OverLoCK-S | 56M | 9.7G | 480 | **84.8%** |\n| OverLoCK-B | 95M | 16.7G | 306 | **85.1%** |\n\n*表：摘自原论文 Table 2 与附录 Table I 的部分行。*\n\n论文摘要强调 OverLoCK-T 用三分之一的参数和 FLOPs 赢过 ConvNeXt-B，吞吐量也比较高，810 对 608，这是真的。\n\n可是别忘了它叫 T。同样叫 T 的 ConvNeXt-T 每秒能跑 1507 张，OverLoCK-T 只有一半多一点。拿差不多快的模型来比比较公平，ConvNeXt-S 每秒 926 张、83.1%，OverLoCK-T 慢一成多，准确率多了 1.1 个百分点。\n\n### 检测与分割\n\n下游任务的差距比分类大。COCO 上用 Mask R-CNN 训练 12 个 epoch，OverLoCK-T 的 box AP 是 48.3，InternImage-T 是 47.2，ConvNeXt-T 是 44.2。ADE20K 上接 UperNet，OverLoCK-T 的单尺度 mIoU 是 50.3，UniRepLKNet-T 是 48.6，ConvNeXt-T 是 46.0。\n\n检测和分割的输入分辨率比分类大得多。ContMix 的区域中心固定是 7×7 个，不管图多大都涵盖全图，这正是它和固定大小的大卷积核拉开差距的地方。\n\n### 一步一步加上去\n\n作者从一个纯静态的卷积网络出发，一次加一样东西。这组实验在 ImageNet 上只训练 120 个 epoch，ADE20K 训练 80K 步，数字和主表不能直接比：\n\n| 模型 | FLOPs | top-1 | mIoU |\n| --- | ---: | ---: | ---: |\n| 静态的基准网络 | 2.6G | 78.5% | 41.1 |\n| 加上循环反馈 | 4.4G | 76.8% | 39.5 |\n| 改用 DDS 三个子网络 | 2.6G | 79.0% | 41.6 |\n| Focus-Net 换成 ContMix 动态区块 | 2.6G | 80.0% | 42.9 |\n| 加上 Overview-Net 的辅助损失 | 2.6G | 80.2% | 43.1 |\n| 每个区块加回最初的 context prior | 2.6G | 80.4% | 43.4 |\n| 加上门控（OverLoCK-XT） | 2.6G | **80.8%** | **43.8** |\n\n*表：摘自原论文 Table 6 的部分行。*\n\n几件事很清楚。\n\n1. **循环的反馈行不通**：照 AbsViT 的方式把第四阶段的输出接回第三阶段，计算量多了七成，准确率反而掉了。\n2. **DDS 在同样计算量下有帮助**：top-1 和 mIoU 各多 0.5，不用多算一轮就有了由上而下的指引。\n3. **最大的一步是 ContMix**：静态区块换成动态区块，top-1 多 1.0，mIoU 多 1.3。\n\n附录还有一组对照。把 Overview-Net 拿掉，改成一般的四阶段金字塔，其他区块不变，top-1 从 80.7% 掉到 79.2%，mIoU 从 43.8 掉到 41.9。由上而下的结构本身值这么多。\n\n### 和其他动态算子比\n\n为了公平比较算子，作者另外搭了一个 Swin 形状的网络，只换 token mixer，卷积核或窗口都是 7×7：\n\n| 算子 | top-1 | ADE20K mIoU |\n| --- | ---: | ---: |\n| DyConv | 77.5% | 36.3 |\n| Involution | 78.3% | 37.5 |\n| 位移窗口注意力（Swin） | 78.4% | 37.4 |\n| 邻域注意力（NATTEN） | 78.6% | 38.1 |\n| DCNv3 | 78.7% | 37.5 |\n| ContMix | **78.8%** | **39.0** |\n\n*表：摘自原论文 Table 7 的部分行。*\n\n分类上大家差不多，ContMix 只比第二名多 0.1；换到分辨率高的分割，就比其他算子多了 0.9 到 2.7。\n\n顺带一提 softmax。ContMix 在产生卷积核时有做 softmax，附录的消融把它拿掉，top-1 从 80.8% 变成 80.5%，mIoU 从 43.8 变成 43.5，影响不大。在 InternImage 里，sigmoid 取代 softmax 会让训练崩掉；DCNv4 则是把 softmax 拿掉之后收敛更快。同样是动态权重，softmax 的分量差很多。\n\n\n> [!TIP]\n> \n> DCNv3 和 DCNv4 的 softmax 之争，可以看这两篇：\n> \n> - **[[22.11] InternImage：大力出奇迹？](/blog/papers/classic-cnns/internimage)**\n> - **[[24.01] DCNv4：权重不设上限](/blog/papers/classic-cnns/dcnv4)**\n\n\n### 实际用起来\n\n官方代码依赖不少外部库。动态卷积最好装 NATTEN，没装也能跑，只是比较慢；大卷积核可以用 UniRepLKNet 那套 iGEMM 实现；检测和分割还要 mmcv。GitHub issue 里有不少安装相关的问题。\n\n大卷积核的加速实现也要看批量大小。有用户[反馈](https://github.com/LMMMEng/OverLoCK/issues/33)打开 `use_gemm` 之后训练反而慢了三倍，作者回复这个实现在小批量时没有优势，批量小就关掉，对准确率没有影响。另一位用户[用 4 张 A100、总批量 8](https://github.com/LMMMEng/OverLoCK/issues/38) 训练 COCO，预估要 41 天；作者自己在 8 张 H800 上是两三天，建议加大批量、开梯度检查点。\n\n## 结论\n\nOverLoCK 的两个想法可以分开看。DDS 让网络先用一段轻量的子网络看完全图，再把这份理解送进后面每个区块；ContMix 让每个位置的卷积核由它和全图区域中心的关系决定，卷积范围还是局部的。前者在同样计算量下多了 0.5，后者多了 1.0 以上，检测和分割的差距又更大。\n\n「先看大局」听起来像是整篇的主角，消融表却显示，出力最多的是 ContMix 把全图信息算进卷积核的那一步。由上而下的结构让这份信息有地方来，ContMix 让它有地方用。\n\n要拿来用的话，先把环境准备好。NATTEN、大卷积核的 CUDA 扩展、mmcv，装齐了再开工；批量小的话，记得把 `use_gemm` 关掉，不然只会更慢。\n","imageSizes":{"/content/papers/assets/d32df0756ccbd0c769259f897b5ad9cf4c8079758897be9a2147cd65a4d46ce1.png":{"width":1047,"height":762},"/content/papers/assets/99458c439c8f5458671d09ec6d68f52f31a20702b6a0bdba99e2fe33dbf86ea7.png":{"width":2112,"height":681}}}