{"notes":[{"id":"classic-cnns/internimage","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InternImage","subtitle":"大力出奇蹟？","author":"Z. Yuan","paperDate":"2022-11-10","publishedOn":"2025-09-04","world":"external/internimage","paperTitle":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","paperUrl":"https://arxiv.org/abs/2211.05778","bibliography":{"title":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","authors":["Wenhai Wang","Jifeng Dai","Zhe Chen","Zhenhang Huang","Zhiqi Li","Xizhou Zhu","Xiaowei Hu","Tong Lu","Lewei Lu","Hongsheng Li","Xiaogang Wang","Yu Qiao"],"url":"https://arxiv.org/abs/2211.05778","pdfUrl":"https://arxiv.org/pdf/2211.05778v4","codeUrl":null,"preprint":{"id":"2211.05778","submittedOn":"2022-11-10","url":"https://arxiv.org/abs/2211.05778"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Wang_InternImage_Exploring_Large-Scale_Vision_Foundation_Models_With_Deformable_Convolutions_CVPR_2023_paper.html"},"citation":"@misc{classiccnnsinternimage2022,\n  title = {{InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions}},\n  author = {Wenhai Wang and Jifeng Dai and Zhe Chen and Zhenhang Huang and Zhiqi Li and Xizhou Zhu and Xiaowei Hu and Tong Lu and Lewei Lu and Hongsheng Li and Xiaogang Wang and Yu Qiao},\n  year = {2022},\n  eprint = {2211.05778},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2211.05778}\n}\n","citationPath":"/content/papers/citations/classic-cnns/internimage.4baca0899bc9.bib"},"thumbnail":{"src":"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png","width":1012,"height":1179},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/internimage.ace4173a8cfd.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"卷積輸在哪裡","text":"卷積輸在哪裡","depth":3},{"id":"dcnv2-為什麼不能直接用","text":"DCNv2 為什麼不能直接用","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"dcnv3-的三個修改","text":"DCNv3 的三個修改","depth":3},{"id":"疊成網路","text":"疊成網路","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"偵測與分割","text":"偵測與分割","depth":3},{"id":"消融softmax-那一項","text":"消融：softmax 那一項","depth":3},{"id":"代價","text":"代價","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/deformable-convnets","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""}]},{"id":"classic-cnns/deformable-convnets","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deformable ConvNets","subtitle":"凌波微步","author":"Z. Yuan","paperDate":"2017-03-17","publishedOn":"2024-05-01","world":"external/deformable-convnets","paperTitle":"Deformable Convolutional Networks","paperUrl":"https://arxiv.org/abs/1703.06211","bibliography":{"title":"Deformable Convolutional Networks","authors":["Jifeng Dai","Haozhi Qi","Yuwen Xiong","Yi Li","Guodong Zhang","Han Hu","Yichen Wei"],"url":"https://arxiv.org/abs/1703.06211","pdfUrl":"https://arxiv.org/pdf/1703.06211v3","codeUrl":null,"preprint":{"id":"1703.06211","submittedOn":"2017-03-17","url":"https://arxiv.org/abs/1703.06211"},"publication":{"venue":"ICCV","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_iccv_2017/html/Dai_Deformable_Convolutional_Networks_ICCV_2017_paper.html"},"citation":"@misc{classiccnnsdeformableconvnets2017,\n  title = {{Deformable Convolutional Networks}},\n  author = {Jifeng Dai and Haozhi Qi and Yuwen Xiong and Yi Li and Guodong Zhang and Han Hu and Yichen Wei},\n  year = {2017},\n  eprint = {1703.06211},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1703.06211}\n}\n","citationPath":"/content/papers/citations/classic-cnns/deformable-convnets.f724f3ff1784.bib"},"thumbnail":{"src":"/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png","width":987,"height":633},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/deformable-convnets.c8fe7262b5e2.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"固定的格子","text":"固定的格子","depth":3},{"id":"已經有人讓網路學變形","text":"已經有人讓網路學變形","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"卷積的兩個步驟","text":"卷積的兩個步驟","depth":3},{"id":"每個點加一個偏移","text":"每個點加一個偏移","depth":3},{"id":"小數位置怎麼取值","text":"小數位置怎麼取值","depth":3},{"id":"偏移從哪裡來","text":"偏移從哪裡來","depth":3},{"id":"兩個特例","text":"兩個特例","depth":3},{"id":"可變形-roi-pooling","text":"可變形 RoI pooling","depth":3},{"id":"放進網路","text":"放進網路","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"感受野真的會跟著物體變","text":"感受野真的會跟著物體變","depth":3},{"id":"要換幾層","text":"要換幾層","depth":3},{"id":"和空洞卷積比","text":"和空洞卷積比","depth":3},{"id":"代價","text":"代價","depth":3},{"id":"coco","text":"COCO","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/stn","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"巨無霸卷積核","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"zh-hant","path":"/content/papers/zh-hant/reparameterization/replknet.16fd94d59060.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"大核卷積使用說明書","text":"大核卷積使用說明書","depth":2},{"id":"深度大核卷積仍然高效","text":"深度大核卷積仍然高效","depth":3},{"id":"大核卷積十分依賴殘差連接","text":"大核卷積十分依賴殘差連接","depth":3},{"id":"大核卷積需要重新參數化","text":"大核卷積需要重新參數化","depth":3},{"id":"大核卷積明顯提升下游任務性能","text":"大核卷積明顯提升下游任務性能","depth":3},{"id":"大核卷積對於小特徵圖的效果依然很好","text":"大核卷積對於小特徵圖的效果依然很好","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"網路架構","text":"網路架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"大核卷積評估","text":"大核卷積評估","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"大核-cnn-比深度小核模型有更大的-erf","text":"大核 CNN 比深度小核模型有更大的 ERF","depth":3},{"id":"形狀偏差","text":"形狀偏差","depth":3},{"id":"限制","text":"限制","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"大卷積核與簡化骨架"}]},{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"卷積核 Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/slak.9c5ac51ae21b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"31-之後","text":"31 之後","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"第一步拆成兩根長條","text":"第一步：拆成兩根長條","depth":3},{"id":"第二步稀疏一點寬一點","text":"第二步：稀疏一點，寬一點","depth":3},{"id":"slak-的樣子","text":"SLaK 的樣子","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任務","text":"下游任務","depth":3},{"id":"實際跑起來","text":"實際跑起來","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]}],"text":"十億參數以上的視覺模型，在 2022 年幾乎清一色是 Transformer。ViT-G 有 18 億參數，SwinV2-G 做到 30 億，BEiT-3 也有 19 億；卷積網路這邊，最大的 ConvNeXt-XL 和 RepLKNet-XL 都停在 3 億多。ConvNeXt 證明了卷積在一般尺寸還能打，可是放到十億參數、上億張影像的規模，還沒有人做出能跟 ViT 比的卷積網路。\n\n這篇來自上海人工智慧實驗室、商湯、清華、南京大學和港中文。作者名單裡的 Jifeng Dai 和 Xizhou Zhu，分別是可變形卷積第一代和第二代的第一作者，這次他們把自家的運算子拿出來，要做成大模型的骨幹。\n\n\n> [!TIP]\n> \n> 可變形卷積可以先看我們的筆記：\n> \n> - **[[17.03] Deformable ConvNets：凌波微步](/blog/papers/classic-cnns/deformable-convnets)**\n\n\n## 定義問題\n\n### 卷積輸在哪裡\n\n作者先比較一般卷積和多頭自注意力（MHSA），整理出兩個差別。\n\n第一是長距離關係。3×3 卷積一層層疊起來，有效感受野其實不大，網路再深也很難像 ViT 那樣一開始就看到整張圖。\n\n第二是自適應的空間聚合。自注意力的權重跟著輸入變，每張圖、每個位置都不一樣；卷積的權重訓練完就固定了，還帶著局部性、平移等變這些很強的歸納偏置。歸納偏置讓卷積網路收斂快、需要的資料少，可是也限制了它從網路規模的資料裡學到更一般的樣式。\n\n[![四格示意圖，比較四種核心運算子，每格中間有一顆藍色星號代表查詢像素。(a) 全域注意力，整張圖都是深淺不一的綠色格子，代表自適應權重，標註有長距離關係、有自適應聚合、但計算與記憶體不省。(b) 局部注意力，只有星號左上方一小塊綠色格子，標註沒有長距離關係。(c) 大卷積核，一大片灰色格子代表固定權重，標註沒有自適應聚合。(d) 作者的動態稀疏卷積，九個綠色小方塊散落在星號四周，遠近不一，三項都打勾](/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png)](/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png)\n\n*原論文 Figure 1，取自 [**v4 PDF 第 1 頁**](https://arxiv.org/pdf/2211.05778v4#page=1)，CC BY 4.0。灰色是固定權重，綠色是隨輸入變化的權重。*\n\nRepLKNet、SLaK 這些大卷積核補上了第一項，第二項還是沒有，權重訓練完照樣固定。可變形卷積兩項都有，每個取樣點的偏移量讓它可以看近也可以看遠，偏移量和調變權重又都是從輸入算出來的。\n\n\n> [!TIP]\n> \n> 大卷積核這條路可以看這兩篇：\n> \n> - **[[22.03] RepLKNet：巨無霸卷積核](/blog/papers/reparameterization/replknet)**\n> - **[[22.07] SLaK：卷積核 Pro Max](/blog/papers/classic-cnns/slak)**\n\n\n### DCNv2 為什麼不能直接用\n\nDCNv2 對位置 $p_0$ 的輸出可以寫成\n\n$$\ny(p_0)=\\sum_{k=1}^{K} w_k\\, m_k\\, x\\!\\left(p_0+p_k+\\Delta p_k\\right),\n$$\n\n$K$ 是取樣點數，3×3 時 $K=9$。$p_k$ 是一般卷積的網格位置，$\\Delta p_k$ 是學出來的偏移量，$m_k$ 是過了 sigmoid 的調變權重，$w_k$ 是第 $k$ 個取樣點的投影權重，大小是 $C\\times C$。\n\n問題在於 DCNv2 平常是拿來當外掛用的，先載入一般卷積預訓練好的權重，再換成可變形卷積微調。要從頭訓練一個十億參數的骨幹，它有兩個毛病。每個取樣點都有自己的一組 $C\\times C$ 權重，參數量和記憶體都跟著 $K$ 成長；sigmoid 讓每個 $m_k$ 落在 0 到 1 之間，九個加起來可以是 0 到 9 之間的任何值，模型一大、資料一多，梯度就不穩。\n\n## 解決問題\n\n### DCNv3 的三個修改\n\n**一、取樣點共用權重。** 借用可分離卷積的想法，把 $w_k$ 拆成兩半。和位置有關的部分交給調變權重 $m_k$，相當於 depthwise；和位置無關的投影 $w$ 由所有取樣點共用，相當於 pointwise。原本九組 $C\\times C$ 的權重剩下一組。\n\n**二、分組。** 學多頭注意力，把通道分成 $G$ 組，每組有自己的偏移量和調變權重，同一層裡不同組可以看不同的地方。\n\n**三、調變權重改用 softmax。** sigmoid 是每個點各算各的，改成九個點一起做 softmax，加總固定是 1。\n\n合起來就是 DCNv3：\n\n$$\ny(p_0)=\\sum_{g=1}^{G}\\sum_{k=1}^{K} w_g\\, m_{gk}\\, x_g\\!\\left(p_0+p_k+\\Delta p_{gk}\\right),\n$$\n\n$x_g$ 是第 $g$ 組的通道，每組 $C' = C/G$ 個通道，$m_{gk}$ 沿著 $K$ 做 softmax。\n\n偏移量是小數，取樣位置通常落在像素之間，所以要用雙線性插值取值。我們用 PyTorch 的 `grid_sample` 寫一個最簡單的版本，先省略最後那個共用的投影：\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ndef dcnv3_core(x, offset, logits, G):\n    # x: (C, H, W)；offset: (G, 9, 2, H, W)，單位是像素；logits: (G, 9, H, W)\n    C, H, W = x.shape\n    m = logits.softmax(dim=1)                                  # 每組 9 個取樣點的權重加起來是 1\n    ys, xs = torch.meshgrid(torch.arange(H), torch.arange(W), indexing=\"ij\")\n    grid = [(dy, dx) for dy in (-1, 0, 1) for dx in (-1, 0, 1)]  # 3×3 的基本位置 p_k\n    out = torch.zeros_like(x)\n    for g, xg in enumerate(x.split(C // G)):                   # 每組有自己的偏移量和權重\n        for k, (dy, dx) in enumerate(grid):\n            py = ys + dy + offset[g, k, 0]                     # p0 + p_k + Δp\n            px = xs + dx + offset[g, k, 1]\n            pos = torch.stack([px / (W - 1) * 2 - 1, py / (H - 1) * 2 - 1], -1)\n            v = F.grid_sample(xg[None], pos[None], align_corners=True)[0]   # 雙線性插值取值\n            out[g * (C // G):(g + 1) * (C // G)] += m[g, k] * v\n    return out                                                 # 之後再接一個所有取樣點共用的線性投影\n\ntorch.manual_seed(0)\nC, H, W, G = 8, 16, 16, 2\nx = torch.randn(C, H, W)\n\n# 偏移量全是 0、權重全部一樣：退化成 3×3 平均\nzero_off, flat = torch.zeros(G, 9, 2, H, W), torch.zeros(G, 9, H, W)\nout = dcnv3_core(x, zero_off, flat, G)\nprint(\"偏移為 0 時，和 3×3 平均池化的最大差距：\", f\"{(out - F.avg_pool2d(x, 3, 1, 1)).abs().max():.1e}\")\n\n# 第 0 組的 9 個點全部往下移 5 格：3×3 的卷積核看到了 5 格以外\noff = zero_off.clone()\noff[0, :, 0] = 5.0\nout = dcnv3_core(x, off, flat, G)\nref = F.avg_pool2d(x, 3, 1, 1)\nprint(\"第 0 組往下移 5 格後，和下方 5 格處平均值的最大差距：\", f\"{(out[:4, :-5] - ref[:4, 5:]).abs().max():.1e}\")\nprint(\"第 1 組不受影響，最大差距：\", f\"{(out[4:] - ref[4:]).abs().max():.1e}\")\n```\n\n```\n偏移為 0 時，和 3×3 平均池化的最大差距： 2.4e-07\n第 0 組往下移 5 格後，和下方 5 格處平均值的最大差距： 1.8e-07\n第 1 組不受影響，最大差距： 2.4e-07\n```\n\n偏移量是 0、softmax 權重平均的時候，DCNv3 就是 3×3 的平均。偏移量一動，同樣九個取樣點就能跑到 5 格外，而且只有那一組跟著動。實際的模型裡，偏移量和調變權重都由一個 3×3 depthwise 卷積接線性層從輸入算出來，所以每個位置都不一樣。官方程式碼裡也有一個純 PyTorch 的版本，註解寫明只用來除錯和測試，訓練時用的是 CUDA 運算子。\n\nsoftmax 那一項看起來只是小修改，我們把兩種做法的加總印出來比一比：\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nfor scale in (0.5, 2.0, 5.0):\n    logits = torch.randn(10000, 9) * scale        # 一萬個位置，每個位置 9 個取樣點\n    s = torch.sigmoid(logits).sum(1)               # DCNv2：每個點各自過 sigmoid\n    t = torch.softmax(logits, 1).sum(1)            # DCNv3：9 個點一起過 softmax\n    print(f\"logit 標準差 {scale}：sigmoid 加總 {s.min():.2f} 到 {s.max():.2f}，softmax 加總 {t.min():.2f} 到 {t.max():.2f}\")\n```\n\n```\nlogit 標準差 0.5：sigmoid 加總 3.07 到 5.87，softmax 加總 1.00 到 1.00\nlogit 標準差 2.0：sigmoid 加總 0.95 到 8.13，softmax 加總 1.00 到 1.00\nlogit 標準差 5.0：sigmoid 加總 0.33 到 8.53，softmax 加總 1.00 到 1.00\n```\n\nsigmoid 的加總在不同位置最多可以差到二十幾倍，等於每個位置的輸出被乘上不同的倍數。softmax 則永遠是 1。等一下的消融實驗會看到，這一項影響有多大。\n\n### 疊成網路\n\n運算子有了，接下來是怎麼用它蓋一個網路。\n\n[![InternImage 架構圖。左邊是主幹，一張小熊貓照片輸入後經過 stem，再依序通過 stage 1 到 stage 4，中間以 downsampling 相接，特徵圖從 H/4×W/4 縮到 H/32×W/32，最後接分類、偵測、分割等任務。右邊三個方框分別寫出 stem 是兩個步長 2 的 3×3 卷積加 LN 與 GELU，downsampling 是步長 2 的 3×3 卷積加 LN；stage i 的基本區塊由 DCNv3、LN、FFN、LN 組成，兩處殘差連接，DCNv3 旁邊標示偏移量 Δp 與調變權重 m；最下方列出四條堆疊規則](/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png)](/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png)\n\n*原論文 Figure 3，取自 [**v4 PDF 第 4 頁**](https://arxiv.org/pdf/2211.05778v4#page=4)，CC BY 4.0。*\n\n基本區塊長得和 Transformer 很像，DCNv3 放在原本自注意力的位置，再接一個用 GELU 的 FFN；兩個分支的輸出都先過 LN，再和殘差相加。stem 和降採樣層則維持卷積網路的做法，stem 是兩個步長 2 的 3×3 卷積，降採樣是一個步長 2 的 3×3 卷積。\n\n四個階段各有通道數 $C_i$、分組數 $G_i$、區塊數 $L_i$，總共 12 個超參數，太多了。作者用四條規則把它壓到 4 個：\n\n1. $C_i = 2^{i-1} C_1$，通道數每個階段翻倍。\n2. $G_i = C_i / C'$，每組的通道數 $C'$ 固定。\n3. $L_1 = L_2 = L_4$。\n4. $L_1 \\le L_3$，第三階段最深。\n\n後兩條就是 ConvNeXt、Swin 那種「AABA」的堆法。剩下 $C_1$、$C'$、$L_1$、$L_3$ 四個數，作者在 3,000 萬參數附近搜了 30 種組合，選出 $(64, 16, 4, 18)$ 當作 InternImage-T。\n\n再往上放大，作者參考 EfficientNet 的複合縮放，深度乘 $\\alpha^{\\phi}$、寬度乘 $\\beta^{\\phi}$，限制 $\\alpha\\beta^{1.99}\\approx 2$，實驗出來 $\\alpha = 1.09$、$\\beta = 1.36$ 最好。最後得到六個大小：\n\n| 模型 | $C_1$ | $C'$ | 各階段區塊數 | 參數量 |\n| --- | ---: | ---: | --- | ---: |\n| InternImage-T | 64 | 16 | 4, 4, 18, 4 | 30M |\n| InternImage-S | 80 | 16 | 4, 4, 21, 4 | 50M |\n| InternImage-B | 112 | 16 | 4, 4, 21, 4 | 97M |\n| InternImage-L | 160 | 16 | 5, 5, 22, 5 | 223M |\n| InternImage-XL | 192 | 16 | 5, 5, 24, 5 | 335M |\n| InternImage-H | 320 | 32 | 6, 6, 32, 6 | 1.08B |\n\n*表：摘自原論文 Table 1。*\n\nT 到 XL 的大小都對齊 ConvNeXt，方便比較；H 是為了衝十億參數另外蓋的。\n\n## 討論\n\n### ImageNet\n\n先看一般尺寸。T、S、B 只用 ImageNet-1K 訓練 300 個 epoch，224×224 輸入：\n\n| 尺寸 | Swin | ConvNeXt | SLaK | HorNet | InternImage |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| T（約 30M） | 81.3% | 82.1% | 82.5% | 83.0% | **83.5%** |\n| S（約 50M） | 83.0% | 83.1% | 83.8% | 84.0% | **84.2%** |\n| B（約 90M） | 83.5% | 83.8% | 84.0% | 84.3% | **84.9%** |\n\n*表：摘自原論文 Table 2 的部分欄位。同一列的模型參數量與 FLOPs 相近。*\n\n同樣的大小，InternImage 每一列都是最高，T 比 ConvNeXt-T 多了 1.4 個百分點，B 也多 1.1。\n\n再看大模型。XL 先在 ImageNet-22K 預訓練，做到 88.0%。H 用 M3I 預訓練在 4.27 億張公開影像上訓練，資料來自 LAION-400M、YFCC-15M 和 CC12M，最後在 640×640 輸入做到 89.6%。最強的 ViT 和混合架構大約是 90.5% 到 90.9%，它們用的是 JFT 這類不公開的資料，差距大約 1 個百分點。\n\n### 偵測與分割\n\n下游任務的差距比分類大。COCO 上用 Mask R-CNN 訓練 12 個 epoch，InternImage-T 的 box AP 是 47.2，Swin-T 是 42.7，ConvNeXt-T 是 44.2，分別多了 4.5 和 3.0。換成 Cascade Mask R-CNN 加多尺度訓練 36 個 epoch，XL 是 56.2，ConvNeXt-XL 是 55.2。ADE20K 語意分割用 UperNet，InternImage-B 是 50.8 mIoU，ConvNeXt-B 49.1，RepLKNet-31B 49.9，SLaK-B 50.2。\n\n論文最常被引用的是 COCO test-dev 上的 65.4 box AP，比之前最好的 FD-SwinV2-G 多 1.2，參數少了 27%。這個數字是怎麼來的值得說一下。骨幹是 InternImage-H，先在 4.27 億張影像上預訓練；再用複合骨幹的技巧把參數加倍成 21.8 億；偵測器換成 DINO，先在 Objects365 上訓練 26 個 epoch，再到 COCO 訓練 12 個 epoch。ADE20K 的 62.9 mIoU 也是 H 配上 Mask2Former，再加多尺度測試。能疊的招都疊上去了，這組數字比較像是在展示這個骨幹撐得起大規模訓練。\n\n\n> [!TIP]\n> \n> DINO 偵測器可以看這篇：\n> \n> - **[[22.03] DINO：靠近，但不夠近](/blog/papers/object-detection/dino-detr)**\n\n\n### 消融：softmax 那一項\n\nDCNv3 的三個修改各拿掉一個，在 InternImage-T 上比較：\n\n| 設定 | ImageNet top-1 | COCO box AP | COCO mask AP |\n| --- | ---: | ---: | ---: |\n| 完整的 DCNv3 | 83.5% | 47.2 | 42.5 |\n| 不共用權重 | 83.6% | 47.4 | 42.6 |\n| 不分組 | 82.3% | 43.8 | 40.0 |\n| sigmoid 取代 softmax | **65.7%** | 38.7 | 35.6 |\n\n*表：摘自原論文 Table 6。COCO 用 Mask R-CNN 訓練 12 個 epoch。*\n\n共用權重幾乎沒有影響準確率，不共用的版本參數卻多了 66.1%。放大到 H 的規模，共用權重能省下 42.0% 的參數和 84.2% 的 GPU 記憶體，這一項是為了把模型做大。\n\n拿掉分組，ImageNet 掉 1.2，COCO 掉 3.4，偵測受的影響比較大。\n\nsoftmax 換回 sigmoid，ImageNet 直接掉到 65.7%，少了將近 18 個百分點。前面印出來的那幾行加總，到這裡就看得出影響了。\n\n作者也把 3×3 換成更大的取樣網格試過。5×5 是 83.6%，參數從 30M 漲到 37M；7×7 反而掉到 82.8%，參數 48M。取樣點會自己移動，3×3 已經夠用。\n\n[![四張同一張羊群照片，分別標示 stage 1 到 stage 4，藍色星號固定在左邊一隻羊的身上。stage 1 和 stage 2 的彩色取樣點都擠在星號附近，旁邊有放大框；stage 3 和 stage 4 的取樣點散布到整張照片，落在其他羊身上和草地上，不同顏色代表不同的組](/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png)](/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png)\n\n*原論文 Figure 5，取自 [**v4 PDF 第 8 頁**](https://arxiv.org/pdf/2211.05778v4#page=8)，CC BY 4.0。藍色星號是查詢點，不同顏色是不同組的取樣位置。*\n\n取樣點的分布也有層次。前兩個階段擠在查詢點附近，看的是局部；到了第三、四階段就散到整張圖。ViT 從第一層就是全域的，這點和它很不一樣。\n\n### 代價\n\n最大的代價是速度。作者在附錄用單張 A100 量了吞吐量，224×224 用批次 256，800×800 用批次 2：\n\n| 模型 | ImageNet top-1 | 224×224（張／秒） | 800×800（張／秒） |\n| --- | ---: | ---: | ---: |\n| ConvNeXt-B | 83.8% | 881 | 58 |\n| RepLKNet-B | 83.5% | 884 | 21 |\n| DAT-B | 84.0% | 661 | 24 |\n| InternImage-B（DCNv2） | — | 311 | 16 |\n| InternImage-B | **84.9%** | 775 | 54 |\n\n*表：摘自原論文 Table 11。*\n\n換成 DCNv3 之後，吞吐量是 DCNv2 版本的 2.5 到 3.4 倍。和 ConvNeXt-B 比還是慢一點，224 時少了 12%，800 時少了 7%；高解析度下倒是 RepLKNet-B 的 2.5 倍，大卷積核在大圖上很吃力。作者在結論也承認，DCN 類運算子的延遲，對講究速度的下游任務還是個問題。\n\n另一個值得看的是資料量。作者從 ImageNet-1K 均勻抽 1%、10% 和全部資料，各訓練 300 個 epoch：\n\n| 模型 | 1% | 10% | 100% |\n| --- | ---: | ---: | ---: |\n| ResNet-50 | **12.2%** | **57.5%** | 80.4% |\n| ConvNeXt-T | 8.4% | 52.6% | 82.1% |\n| Swin-T | 失敗 | 12.1% | 81.3% |\n| InternImage-T | 5.9% | 56.0% | **83.5%** |\n\n*表：摘自原論文 Table 12。*\n\n作者說 InternImage-T 在 1% 和 10% 資料上都很穩健。可是 1% 那一欄它輸給 ResNet-50 和 ConvNeXt-T，只贏了直接訓練失敗的 Swin-T；10% 也還是輸 ResNet-50。它比 Swin-T 耐得住少量資料，這倒是真的；資料少到只剩 1% 時，歸納偏置比較強的 ResNet-50 和 ConvNeXt-T 還是比較好。\n\n## 結論\n\n破紀錄的那幾個數字，確實是大力出來的。4.27 億張影像、參數加倍的複合骨幹、Objects365 預訓練，再配上 DINO，才有 COCO 的 65.4。\n\n可是把尺寸拉回一般大小，參數和 FLOPs 都和 ConvNeXt 對齊，InternImage 在 ImageNet 還是多了 1 個百分點以上，COCO 的 box AP 多了 1.8 到 3.0。這一段沒有大力可以借，靠的是 DCNv3 本身。\n\n想自己跑的話，得先和那個自訂的 CUDA 運算子打好關係，編譯、部署、匯出 TensorRT，每一關都要先過它。\n\n最後說個小插曲。這篇的消融實驗裡，softmax 換回 sigmoid，準確率就從 83.5% 摔到 65.7%。一年多後，大致是同一群人推出了 DCNv4，兩項主要改動之一，就是把 softmax 整個拿掉，結果還收斂得更快。\n\n論文裡的關鍵設計，保鮮期有時候只有一年。\n","imageSizes":{"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png":{"width":1012,"height":1179},"/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png":{"width":1017,"height":1467},"/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png":{"width":1012,"height":666}}}