{"notes":[{"id":"classic-cnns/internimage","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InternImage","subtitle":"大力出奇迹？","author":"Z. Yuan","paperDate":"2022-11-10","publishedOn":"2025-09-04","world":"external/internimage","paperTitle":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","paperUrl":"https://arxiv.org/abs/2211.05778","bibliography":{"title":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","authors":["Wenhai Wang","Jifeng Dai","Zhe Chen","Zhenhang Huang","Zhiqi Li","Xizhou Zhu","Xiaowei Hu","Tong Lu","Lewei Lu","Hongsheng Li","Xiaogang Wang","Yu Qiao"],"url":"https://arxiv.org/abs/2211.05778","pdfUrl":"https://arxiv.org/pdf/2211.05778v4","codeUrl":null,"preprint":{"id":"2211.05778","submittedOn":"2022-11-10","url":"https://arxiv.org/abs/2211.05778"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Wang_InternImage_Exploring_Large-Scale_Vision_Foundation_Models_With_Deformable_Convolutions_CVPR_2023_paper.html"},"citation":"@misc{classiccnnsinternimage2022,\n  title = {{InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions}},\n  author = {Wenhai Wang and Jifeng Dai and Zhe Chen and Zhenhang Huang and Zhiqi Li and Xizhou Zhu and Xiaowei Hu and Tong Lu and Lewei Lu and Hongsheng Li and Xiaogang Wang and Yu Qiao},\n  year = {2022},\n  eprint = {2211.05778},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2211.05778}\n}\n","citationPath":"/content/papers/citations/classic-cnns/internimage.4baca0899bc9.bib"},"thumbnail":{"src":"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png","width":1012,"height":1179},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/internimage.f5dfee837064.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"卷积输在哪里","text":"卷积输在哪里","depth":3},{"id":"dcnv2-为什么不能直接用","text":"DCNv2 为什么不能直接用","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"dcnv3-的三个修改","text":"DCNv3 的三个修改","depth":3},{"id":"叠成网络","text":"叠成网络","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"检测与分割","text":"检测与分割","depth":3},{"id":"消融softmax-那一项","text":"消融：softmax 那一项","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/deformable-convnets","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""}]},{"id":"classic-cnns/deformable-convnets","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deformable ConvNets","subtitle":"凌波微步","author":"Z. Yuan","paperDate":"2017-03-17","publishedOn":"2024-05-01","world":"external/deformable-convnets","paperTitle":"Deformable Convolutional Networks","paperUrl":"https://arxiv.org/abs/1703.06211","bibliography":{"title":"Deformable Convolutional Networks","authors":["Jifeng Dai","Haozhi Qi","Yuwen Xiong","Yi Li","Guodong Zhang","Han Hu","Yichen Wei"],"url":"https://arxiv.org/abs/1703.06211","pdfUrl":"https://arxiv.org/pdf/1703.06211v3","codeUrl":null,"preprint":{"id":"1703.06211","submittedOn":"2017-03-17","url":"https://arxiv.org/abs/1703.06211"},"publication":{"venue":"ICCV","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_iccv_2017/html/Dai_Deformable_Convolutional_Networks_ICCV_2017_paper.html"},"citation":"@misc{classiccnnsdeformableconvnets2017,\n  title = {{Deformable Convolutional Networks}},\n  author = {Jifeng Dai and Haozhi Qi and Yuwen Xiong and Yi Li and Guodong Zhang and Han Hu and Yichen Wei},\n  year = {2017},\n  eprint = {1703.06211},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1703.06211}\n}\n","citationPath":"/content/papers/citations/classic-cnns/deformable-convnets.f724f3ff1784.bib"},"thumbnail":{"src":"/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png","width":987,"height":633},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/deformable-convnets.b75c34ef240f.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"固定的网格","text":"固定的网格","depth":3},{"id":"已经有人让网络学变形","text":"已经有人让网络学变形","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"卷积的两个步骤","text":"卷积的两个步骤","depth":3},{"id":"每个点加一个偏移量","text":"每个点加一个偏移量","depth":3},{"id":"小数位置怎么取值","text":"小数位置怎么取值","depth":3},{"id":"偏移量从哪里来","text":"偏移量从哪里来","depth":3},{"id":"两个特例","text":"两个特例","depth":3},{"id":"可变形-roi-pooling","text":"可变形 RoI pooling","depth":3},{"id":"放进网络","text":"放进网络","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"感受野真的会跟着物体变","text":"感受野真的会跟着物体变","depth":3},{"id":"要换几层","text":"要换几层","depth":3},{"id":"和空洞卷积比","text":"和空洞卷积比","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"coco","text":"COCO","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/stn","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"巨无霸卷积核","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"zh-hans","path":"/content/papers/zh-hans/reparameterization/replknet.63f1e846afbb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"大核卷积使用说明书","text":"大核卷积使用说明书","depth":2},{"id":"深度大核卷积仍然高效","text":"深度大核卷积仍然高效","depth":3},{"id":"大核卷积十分依赖残差连接","text":"大核卷积十分依赖残差连接","depth":3},{"id":"大核卷积需要重参数化","text":"大核卷积需要重参数化","depth":3},{"id":"大核卷积明显提升下游任务性能","text":"大核卷积明显提升下游任务性能","depth":3},{"id":"大核卷积对于小特征图的效果依然很好","text":"大核卷积对于小特征图的效果依然很好","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"网络架构","text":"网络架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"大核卷积评估","text":"大核卷积评估","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"大核-cnn-比深度小核模型有更大的-erf","text":"大核 CNN 比深度小核模型有更大的 ERF","depth":3},{"id":"形状偏差","text":"形状偏差","depth":3},{"id":"限制","text":"限制","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"大卷积核与简化骨架"}]},{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"卷积核 Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/slak.d84cfaf376c9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"31-之后","text":"31 之后","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"第一步拆成两根长条","text":"第一步：拆成两根长条","depth":3},{"id":"第二步稀疏一点宽一点","text":"第二步：稀疏一点，宽一点","depth":3},{"id":"slak-的样子","text":"SLaK 的样子","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任务","text":"下游任务","depth":3},{"id":"实际跑起来","text":"实际跑起来","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]}],"text":"十亿参数以上的视觉模型，在 2022 年几乎清一色是 Transformer。ViT-G 有 18 亿参数，SwinV2-G 做到 30 亿，BEiT-3 也有 19 亿；卷积网络这边，最大的 ConvNeXt-XL 和 RepLKNet-XL 都停在 3 亿多。ConvNeXt 证明了卷积在一般尺寸上还能打，可是放到十亿参数、上亿张图像的规模，还没有人做出能跟 ViT 比的卷积网络。\n\n这篇来自上海人工智能实验室、商汤、清华、南京大学和港中文。作者名单里的 Jifeng Dai 和 Xizhou Zhu，分别是可变形卷积第一代和第二代的第一作者，这次他们把自家的算子拿出来，要做成大模型的骨干。\n\n\n> [!TIP]\n> \n> 可变形卷积可以先看我们的笔记：\n> \n> - **[[17.03] Deformable ConvNets：凌波微步](/blog/papers/classic-cnns/deformable-convnets)**\n\n\n## 定义问题\n\n### 卷积输在哪里\n\n作者先比较一般卷积和多头自注意力（MHSA），整理出两个差别。\n\n第一是长距离关系。3×3 卷积一层层叠起来，有效感受野其实不大，网络再深也很难像 ViT 那样一开始就看到整张图。\n\n第二是自适应的空间聚合。自注意力的权重跟着输入变，每张图、每个位置都不一样；卷积的权重训练完就固定了，还带着局部性、平移等变这些很强的归纳偏置。归纳偏置让卷积网络收敛快、需要的数据少，可是也限制了它从互联网规模的数据里学到更一般的模式。\n\n[![四格示意图，比较四种核心算子，每格中间有一颗蓝色星号代表查询像素。(a) 全局注意力，整张图都是深浅不一的绿色格子，代表自适应权重，标注有长距离关系、有自适应聚合，但计算与内存不省。(b) 局部注意力，只有星号左上方一小块绿色格子，标注没有长距离关系。(c) 大卷积核，一大片灰色格子代表固定权重，标注没有自适应聚合。(d) 作者的动态稀疏卷积，九个绿色小方块散落在星号四周，远近不一，三项都打勾](/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png)](/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png)\n\n*原论文 Figure 1，取自 [**v4 PDF 第 1 页**](https://arxiv.org/pdf/2211.05778v4#page=1)，CC BY 4.0。灰色是固定权重，绿色是随输入变化的权重。*\n\nRepLKNet、SLaK 这些大卷积核补上了第一项，第二项还是没有，权重训练完照样固定。可变形卷积两项都有，每个采样点的偏移量让它可以看近也可以看远，偏移量和调制权重又都是从输入算出来的。\n\n\n> [!TIP]\n> \n> 大卷积核这条路可以看这两篇：\n> \n> - **[[22.03] RepLKNet：巨无霸卷积核](/blog/papers/reparameterization/replknet)**\n> - **[[22.07] SLaK：卷积核 Pro Max](/blog/papers/classic-cnns/slak)**\n\n\n### DCNv2 为什么不能直接用\n\nDCNv2 对位置 $p_0$ 的输出可以写成\n\n$$\ny(p_0)=\\sum_{k=1}^{K} w_k\\, m_k\\, x\\!\\left(p_0+p_k+\\Delta p_k\\right),\n$$\n\n$K$ 是采样点数，3×3 时 $K=9$。$p_k$ 是一般卷积的网格位置，$\\Delta p_k$ 是学出来的偏移量，$m_k$ 是过了 sigmoid 的调制权重，$w_k$ 是第 $k$ 个采样点的投影权重，大小是 $C\\times C$。\n\n问题在于 DCNv2 平常是拿来当外挂用的，先加载一般卷积预训练好的权重，再换成可变形卷积微调。要从头训练一个十亿参数的骨干，它有两个毛病。每个采样点都有自己的一组 $C\\times C$ 权重，参数量和内存都跟着 $K$ 增长；sigmoid 让每个 $m_k$ 落在 0 到 1 之间，九个加起来可以是 0 到 9 之间的任何值，模型一大、数据一多，梯度就不稳。\n\n## 解决问题\n\n### DCNv3 的三个修改\n\n**一、采样点共享权重**。借用可分离卷积的想法，把 $w_k$ 拆成两半。和位置有关的部分交给调制权重 $m_k$，相当于 depthwise；和位置无关的投影 $w$ 由所有采样点共享，相当于 pointwise。原本九组 $C\\times C$ 的权重只剩下一组。\n\n**二、分组**。学多头注意力，把通道分成 $G$ 组，每组有自己的偏移量和调制权重，同一层里不同组可以看不同的地方。\n\n**三、调制权重改用 softmax**。sigmoid 是每个点各算各的，改成九个点一起做 softmax，加总固定是 1。\n\n合起来就是 DCNv3：\n\n$$\ny(p_0)=\\sum_{g=1}^{G}\\sum_{k=1}^{K} w_g\\, m_{gk}\\, x_g\\!\\left(p_0+p_k+\\Delta p_{gk}\\right),\n$$\n\n$x_g$ 是第 $g$ 组的通道，每组 $C' = C/G$ 个通道，$m_{gk}$ 沿着 $K$ 做 softmax。\n\n偏移量是小数，采样位置通常落在像素之间，所以要用双线性插值取值。我们用 PyTorch 的 `grid_sample` 写一个最简单的版本，先省略最后那个共享的投影：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ndef dcnv3_core(x, offset, logits, G):\n    # x: (C, H, W)；offset: (G, 9, 2, H, W)，单位是像素；logits: (G, 9, H, W)\n    C, H, W = x.shape\n    m = logits.softmax(dim=1)                                  # 每组 9 个采样点的权重加起来是 1\n    ys, xs = torch.meshgrid(torch.arange(H), torch.arange(W), indexing=\"ij\")\n    grid = [(dy, dx) for dy in (-1, 0, 1) for dx in (-1, 0, 1)]  # 3×3 的基本位置 p_k\n    out = torch.zeros_like(x)\n    for g, xg in enumerate(x.split(C // G)):                   # 每组有自己的偏移量和权重\n        for k, (dy, dx) in enumerate(grid):\n            py = ys + dy + offset[g, k, 0]                     # p0 + p_k + Δp\n            px = xs + dx + offset[g, k, 1]\n            pos = torch.stack([px / (W - 1) * 2 - 1, py / (H - 1) * 2 - 1], -1)\n            v = F.grid_sample(xg[None], pos[None], align_corners=True)[0]   # 双线性插值取值\n            out[g * (C // G):(g + 1) * (C // G)] += m[g, k] * v\n    return out                                                 # 之后再接一个所有采样点共享的线性投影\n\ntorch.manual_seed(0)\nC, H, W, G = 8, 16, 16, 2\nx = torch.randn(C, H, W)\n\n# 偏移量全是 0、权重全部一样：退化成 3×3 平均\nzero_off, flat = torch.zeros(G, 9, 2, H, W), torch.zeros(G, 9, H, W)\nout = dcnv3_core(x, zero_off, flat, G)\nprint(\"偏移为 0 时，和 3×3 平均池化的最大差距：\", f\"{(out - F.avg_pool2d(x, 3, 1, 1)).abs().max():.1e}\")\n\n# 第 0 组的 9 个点全部往下移 5 格：3×3 的卷积核看到了 5 格以外\noff = zero_off.clone()\noff[0, :, 0] = 5.0\nout = dcnv3_core(x, off, flat, G)\nref = F.avg_pool2d(x, 3, 1, 1)\nprint(\"第 0 组往下移 5 格后，和下方 5 格处平均值的最大差距：\", f\"{(out[:4, :-5] - ref[:4, 5:]).abs().max():.1e}\")\nprint(\"第 1 组不受影响，最大差距：\", f\"{(out[4:] - ref[4:]).abs().max():.1e}\")\n```\n\n```\n偏移为 0 时，和 3×3 平均池化的最大差距： 2.4e-07\n第 0 组往下移 5 格后，和下方 5 格处平均值的最大差距： 1.8e-07\n第 1 组不受影响，最大差距： 2.4e-07\n```\n\n偏移量是 0、softmax 权重均匀的时候，DCNv3 就是 3×3 的平均。偏移量一动，同样九个采样点就能跑到 5 格外，而且只有那一组跟着动。实际的模型里，偏移量和调制权重都由一个 3×3 depthwise 卷积接线性层从输入算出来，所以每个位置都不一样。官方代码里也有一个纯 PyTorch 的版本，注释写明只用于调试和测试，训练时用的是 CUDA 算子。\n\nsoftmax 那一项看起来只是小修改，我们把两种做法的加总打印出来比一比：\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nfor scale in (0.5, 2.0, 5.0):\n    logits = torch.randn(10000, 9) * scale        # 一万个位置，每个位置 9 个采样点\n    s = torch.sigmoid(logits).sum(1)               # DCNv2：每个点各自过 sigmoid\n    t = torch.softmax(logits, 1).sum(1)            # DCNv3：9 个点一起过 softmax\n    print(f\"logit 标准差 {scale}：sigmoid 加总 {s.min():.2f} 到 {s.max():.2f}，softmax 加总 {t.min():.2f} 到 {t.max():.2f}\")\n```\n\n```\nlogit 标准差 0.5：sigmoid 加总 3.07 到 5.87，softmax 加总 1.00 到 1.00\nlogit 标准差 2.0：sigmoid 加总 0.95 到 8.13，softmax 加总 1.00 到 1.00\nlogit 标准差 5.0：sigmoid 加总 0.33 到 8.53，softmax 加总 1.00 到 1.00\n```\n\nsigmoid 的加总在不同位置最多可以差到二十几倍，相当于每个位置的输出被乘上不同的倍数。softmax 则永远是 1。等一下的消融实验会看到，这一项影响有多大。\n\n### 叠成网络\n\n算子有了，接下来是怎么用它搭一个网络。\n\n[![InternImage 架构图。左边是主干，一张小熊猫照片输入后经过 stem，再依次通过 stage 1 到 stage 4，中间以 downsampling 相接，特征图从 H/4×W/4 缩到 H/32×W/32，最后接分类、检测、分割等任务。右边三个方框分别写出 stem 是两个步长 2 的 3×3 卷积加 LN 与 GELU，downsampling 是步长 2 的 3×3 卷积加 LN；stage i 的基本块由 DCNv3、LN、FFN、LN 组成，两处残差连接，DCNv3 旁边标示偏移量 Δp 与调制权重 m；最下方列出四条堆叠规则](/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png)](/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png)\n\n*原论文 Figure 3，取自 [**v4 PDF 第 4 页**](https://arxiv.org/pdf/2211.05778v4#page=4)，CC BY 4.0。*\n\n基本块长得和 Transformer 很像，DCNv3 放在原本自注意力的位置，再接一个用 GELU 的 FFN；两个分支的输出都先过 LN，再和残差相加。stem 和降采样层则维持卷积网络的做法，stem 是两个步长 2 的 3×3 卷积，降采样是一个步长 2 的 3×3 卷积。\n\n四个阶段各有通道数 $C_i$、分组数 $G_i$、块数 $L_i$，总共 12 个超参数，太多了。作者用四条规则把它压到 4 个：\n\n1. $C_i = 2^{i-1} C_1$，通道数每个阶段翻倍。\n2. $G_i = C_i / C'$，每组的通道数 $C'$ 固定。\n3. $L_1 = L_2 = L_4$。\n4. $L_1 \\le L_3$，第三阶段最深。\n\n后两条就是 ConvNeXt、Swin 那种「AABA」的堆法。剩下 $C_1$、$C'$、$L_1$、$L_3$ 四个数，作者在 3,000 万参数附近搜了 30 种组合，选出 $(64, 16, 4, 18)$ 作为 InternImage-T。\n\n再往上放大，作者参考 EfficientNet 的复合缩放，深度乘 $\\alpha^{\\phi}$、宽度乘 $\\beta^{\\phi}$，限制 $\\alpha\\beta^{1.99}\\approx 2$，实验得出 $\\alpha = 1.09$、$\\beta = 1.36$ 最好。最后得到六个大小：\n\n| 模型 | $C_1$ | $C'$ | 各阶段块数 | 参数量 |\n| --- | ---: | ---: | --- | ---: |\n| InternImage-T | 64 | 16 | 4, 4, 18, 4 | 30M |\n| InternImage-S | 80 | 16 | 4, 4, 21, 4 | 50M |\n| InternImage-B | 112 | 16 | 4, 4, 21, 4 | 97M |\n| InternImage-L | 160 | 16 | 5, 5, 22, 5 | 223M |\n| InternImage-XL | 192 | 16 | 5, 5, 24, 5 | 335M |\n| InternImage-H | 320 | 32 | 6, 6, 32, 6 | 1.08B |\n\n*表：摘自原论文 Table 1。*\n\nT 到 XL 的大小都对齐 ConvNeXt，方便比较；H 是为了冲十亿参数另外搭的。\n\n## 讨论\n\n### ImageNet\n\n先看一般尺寸。T、S、B 只用 ImageNet-1K 训练 300 个 epoch，224×224 输入：\n\n| 尺寸 | Swin | ConvNeXt | SLaK | HorNet | InternImage |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| T（约 30M） | 81.3% | 82.1% | 82.5% | 83.0% | **83.5%** |\n| S（约 50M） | 83.0% | 83.1% | 83.8% | 84.0% | **84.2%** |\n| B（约 90M） | 83.5% | 83.8% | 84.0% | 84.3% | **84.9%** |\n\n*表：摘自原论文 Table 2 的部分列。同一行的模型参数量与 FLOPs 相近。*\n\n同样的大小，InternImage 每一行都是最高，T 比 ConvNeXt-T 多了 1.4 个百分点，B 也多了 1.1。\n\n再看大模型。XL 先在 ImageNet-22K 上预训练，做到 88.0%。H 用 M3I 预训练在 4.27 亿张公开图像上训练，数据来自 LAION-400M、YFCC-15M 和 CC12M，最后在 640×640 输入下做到 89.6%。最强的 ViT 和混合架构大约是 90.5% 到 90.9%，它们用的是 JFT 这类不公开的数据，差距大约 1 个百分点。\n\n### 检测与分割\n\n下游任务的差距比分类大。COCO 上用 Mask R-CNN 训练 12 个 epoch，InternImage-T 的 box AP 是 47.2，Swin-T 是 42.7，ConvNeXt-T 是 44.2，分别多了 4.5 和 3.0。换成 Cascade Mask R-CNN 加多尺度训练 36 个 epoch，XL 是 56.2，ConvNeXt-XL 是 55.2。ADE20K 语义分割用 UperNet，InternImage-B 是 50.8 mIoU，ConvNeXt-B 49.1，RepLKNet-31B 49.9，SLaK-B 50.2。\n\n论文最常被引用的是 COCO test-dev 上的 65.4 box AP，比之前最好的 FD-SwinV2-G 多 1.2，参数少了 27%。这个数字是怎么来的，值得说一下。骨干是 InternImage-H，先在 4.27 亿张图像上预训练；再用复合骨干的技巧把参数加倍成 21.8 亿；检测器换成 DINO，先在 Objects365 上训练 26 个 epoch，再到 COCO 训练 12 个 epoch。ADE20K 的 62.9 mIoU 也是 H 配上 Mask2Former，再加多尺度测试。能叠的招都叠上去了，这组数字比较像是在展示这个骨干撑得起大规模训练。\n\n\n> [!TIP]\n> \n> DINO 检测器可以看这篇：\n> \n> - **[[22.03] DINO：靠近，但不够近](/blog/papers/object-detection/dino-detr)**\n\n\n### 消融：softmax 那一项\n\nDCNv3 的三个修改各拿掉一个，在 InternImage-T 上比较：\n\n| 设置 | ImageNet top-1 | COCO box AP | COCO mask AP |\n| --- | ---: | ---: | ---: |\n| 完整的 DCNv3 | 83.5% | 47.2 | 42.5 |\n| 不共享权重 | 83.6% | 47.4 | 42.6 |\n| 不分组 | 82.3% | 43.8 | 40.0 |\n| sigmoid 取代 softmax | **65.7%** | 38.7 | 35.6 |\n\n*表：摘自原论文 Table 6。COCO 用 Mask R-CNN 训练 12 个 epoch。*\n\n共享权重几乎没有影响准确率，不共享的版本参数却多了 66.1%。放大到 H 的规模，共享权重能省下 42.0% 的参数和 84.2% 的 GPU 显存，这一项是为了把模型做大。\n\n拿掉分组，ImageNet 掉 1.2，COCO 掉 3.4，检测受的影响比较大。\n\nsoftmax 换回 sigmoid，ImageNet 直接掉到 65.7%，少了将近 18 个百分点。前面打印出来的那几行加总，到这里就看得出影响了。\n\n作者也试过把 3×3 换成更大的采样网格。5×5 是 83.6%，参数从 30M 涨到 37M；7×7 反而掉到 82.8%，参数 48M。采样点会自己移动，3×3 已经够用。\n\n[![四张相同的羊群照片，分别标示 stage 1 到 stage 4，蓝色星号固定在左边一只羊的身上。stage 1 和 stage 2 的彩色采样点都挤在星号附近，旁边有放大框；stage 3 和 stage 4 的采样点散布到整张照片，落在其他羊身上和草地上，不同颜色代表不同的组](/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png)](/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png)\n\n*原论文 Figure 5，取自 [**v4 PDF 第 8 页**](https://arxiv.org/pdf/2211.05778v4#page=8)，CC BY 4.0。蓝色星号是查询点，不同颜色是不同组的采样位置。*\n\n采样点的分布也有层次。前两个阶段挤在查询点附近，看的是局部；到了第三、四阶段就散到整张图。ViT 从第一层就是全局的，这点和它很不一样。\n\n### 代价\n\n最大的代价是速度。作者在附录里用单张 A100 测了吞吐量，224×224 用批量 256，800×800 用批量 2：\n\n| 模型 | ImageNet top-1 | 224×224（张/秒） | 800×800（张/秒） |\n| --- | ---: | ---: | ---: |\n| ConvNeXt-B | 83.8% | 881 | 58 |\n| RepLKNet-B | 83.5% | 884 | 21 |\n| DAT-B | 84.0% | 661 | 24 |\n| InternImage-B（DCNv2） | — | 311 | 16 |\n| InternImage-B | **84.9%** | 775 | 54 |\n\n*表：摘自原论文 Table 11。*\n\n换成 DCNv3 之后，吞吐量是 DCNv2 版本的 2.5 到 3.4 倍。和 ConvNeXt-B 比还是慢一点，224 时少了 12%，800 时少了 7%；高分辨率下倒是 RepLKNet-B 的 2.5 倍，大卷积核在大图上很吃力。作者在结论里也承认，DCN 类算子的延迟，对讲究速度的下游任务仍是个问题。\n\n另一个值得看的是数据量。作者从 ImageNet-1K 均匀抽取 1%、10% 和全部数据，各训练 300 个 epoch：\n\n| 模型 | 1% | 10% | 100% |\n| --- | ---: | ---: | ---: |\n| ResNet-50 | **12.2%** | **57.5%** | 80.4% |\n| ConvNeXt-T | 8.4% | 52.6% | 82.1% |\n| Swin-T | 失败 | 12.1% | 81.3% |\n| InternImage-T | 5.9% | 56.0% | **83.5%** |\n\n*表：摘自原论文 Table 12。*\n\n作者说 InternImage-T 在 1% 和 10% 数据上都很稳健。可是 1% 那一列它输给了 ResNet-50 和 ConvNeXt-T，只赢了直接训练失败的 Swin-T；10% 也还是输给 ResNet-50。它比 Swin-T 扛得住少量数据，这倒是真的；数据少到只剩 1% 时，归纳偏置比较强的 ResNet-50 和 ConvNeXt-T 还是比较好。\n\n## 结论\n\n破纪录的那几个数字，确实是大力出来的。4.27 亿张图像、参数加倍的复合骨干、Objects365 预训练，再配上 DINO，才有 COCO 的 65.4。\n\n可是把尺寸拉回一般大小，参数和 FLOPs 都和 ConvNeXt 对齐，InternImage 在 ImageNet 上还是多了 1 个百分点以上，COCO 的 box AP 多了 1.8 到 3.0。这一段没有大力可以借，靠的是 DCNv3 本身。\n\n想自己跑的话，得先和那个自定义的 CUDA 算子搞好关系，编译、部署、导出 TensorRT，每一关都要先过它。\n\n最后说个小插曲。这篇的消融实验里，softmax 换回 sigmoid，准确率就从 83.5% 摔到 65.7%。一年多后，大致是同一群人推出了 DCNv4，两项主要改动之一，就是把 softmax 整个拿掉，结果还收敛得更快。\n\n论文里的关键设计，保鲜期有时候只有一年。\n","imageSizes":{"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png":{"width":1012,"height":1179},"/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png":{"width":1017,"height":1467},"/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png":{"width":1012,"height":666}}}