{"notes":[{"id":"classic-cnns/internimage","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InternImage","subtitle":"Miracles by Brute Force?","author":"Z. Yuan","paperDate":"2022-11-10","publishedOn":"2025-09-04","world":"external/internimage","paperTitle":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","paperUrl":"https://arxiv.org/abs/2211.05778","bibliography":{"title":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","authors":["Wenhai Wang","Jifeng Dai","Zhe Chen","Zhenhang Huang","Zhiqi Li","Xizhou Zhu","Xiaowei Hu","Tong Lu","Lewei Lu","Hongsheng Li","Xiaogang Wang","Yu Qiao"],"url":"https://arxiv.org/abs/2211.05778","pdfUrl":"https://arxiv.org/pdf/2211.05778v4","codeUrl":null,"preprint":{"id":"2211.05778","submittedOn":"2022-11-10","url":"https://arxiv.org/abs/2211.05778"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Wang_InternImage_Exploring_Large-Scale_Vision_Foundation_Models_With_Deformable_Convolutions_CVPR_2023_paper.html"},"citation":"@misc{classiccnnsinternimage2022,\n  title = {{InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions}},\n  author = {Wenhai Wang and Jifeng Dai and Zhe Chen and Zhenhang Huang and Zhiqi Li and Xizhou Zhu and Xiaowei Hu and Tong Lu and Lewei Lu and Hongsheng Li and Xiaogang Wang and Yu Qiao},\n  year = {2022},\n  eprint = {2211.05778},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2211.05778}\n}\n","citationPath":"/content/papers/citations/classic-cnns/internimage.4baca0899bc9.bib"},"thumbnail":{"src":"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png","width":1012,"height":1179},"locale":"en","path":"/content/papers/en/classic-cnns/internimage.02b85ef97de6.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"where-convolutions-fall-short","text":"Where Convolutions Fall Short","depth":3},{"id":"why-dcnv2-cant-be-used-as-is","text":"Why DCNv2 Can't Be Used As-Is","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"dcnv3s-three-changes","text":"DCNv3's Three Changes","depth":3},{"id":"stacking-it-into-a-network","text":"Stacking It into a Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"detection-and-segmentation","text":"Detection and Segmentation","depth":3},{"id":"ablation-the-softmax-change","text":"Ablation: The Softmax Change","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/deformable-convnets","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""}]},{"id":"classic-cnns/deformable-convnets","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deformable ConvNets","subtitle":"Fancy Footwork","author":"Z. Yuan","paperDate":"2017-03-17","publishedOn":"2024-05-01","world":"external/deformable-convnets","paperTitle":"Deformable Convolutional Networks","paperUrl":"https://arxiv.org/abs/1703.06211","bibliography":{"title":"Deformable Convolutional Networks","authors":["Jifeng Dai","Haozhi Qi","Yuwen Xiong","Yi Li","Guodong Zhang","Han Hu","Yichen Wei"],"url":"https://arxiv.org/abs/1703.06211","pdfUrl":"https://arxiv.org/pdf/1703.06211v3","codeUrl":null,"preprint":{"id":"1703.06211","submittedOn":"2017-03-17","url":"https://arxiv.org/abs/1703.06211"},"publication":{"venue":"ICCV","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_iccv_2017/html/Dai_Deformable_Convolutional_Networks_ICCV_2017_paper.html"},"citation":"@misc{classiccnnsdeformableconvnets2017,\n  title = {{Deformable Convolutional Networks}},\n  author = {Jifeng Dai and Haozhi Qi and Yuwen Xiong and Yi Li and Guodong Zhang and Han Hu and Yichen Wei},\n  year = {2017},\n  eprint = {1703.06211},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1703.06211}\n}\n","citationPath":"/content/papers/citations/classic-cnns/deformable-convnets.f724f3ff1784.bib"},"thumbnail":{"src":"/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png","width":987,"height":633},"locale":"en","path":"/content/papers/en/classic-cnns/deformable-convnets.319cd37ba4c1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-fixed-grid","text":"The Fixed Grid","depth":3},{"id":"someone-already-taught-networks-to-warp","text":"Someone Already Taught Networks to Warp","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-two-steps-of-convolution","text":"The Two Steps of Convolution","depth":3},{"id":"an-offset-for-every-point","text":"An Offset for Every Point","depth":3},{"id":"sampling-at-fractional-positions","text":"Sampling at Fractional Positions","depth":3},{"id":"where-the-offsets-come-from","text":"Where the Offsets Come From","depth":3},{"id":"two-special-cases","text":"Two Special Cases","depth":3},{"id":"deformable-roi-pooling","text":"Deformable RoI Pooling","depth":3},{"id":"into-the-network","text":"Into the Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"receptive-fields-really-do-follow-the-object","text":"Receptive Fields Really Do Follow the Object","depth":3},{"id":"how-many-layers-to-replace","text":"How Many Layers to Replace","depth":3},{"id":"compared-with-atrous-convolution","text":"Compared with Atrous Convolution","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"coco","text":"COCO","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/stn","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""}]},{"id":"reparameterization/replknet","category":"reparameterization","legacyPath":"reparameterization/2203-replknet","legacySlug":"2203-replknet","legacyPaths":[],"title":"RepLKNet","subtitle":"Giant Convolutional Kernels","author":"Z. Yuan","paperDate":"2022-03-13","publishedOn":"2024-07-21","paperTitle":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","paperUrl":"https://arxiv.org/abs/2203.06717","bibliography":{"title":"Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs","authors":["Xiaohan Ding","Xiangyu Zhang","Yizhuang Zhou","Jungong Han","Guiguang Ding","Jian Sun"],"url":"https://arxiv.org/abs/2203.06717","pdfUrl":"https://arxiv.org/pdf/2203.06717","codeUrl":null,"preprint":{"id":"2203.06717","submittedOn":"2022-03-13","url":"https://arxiv.org/abs/2203.06717"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.06717"},"citation":"@misc{reparameterizationreplknet2022,\n  title = {{Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs}},\n  author = {Xiaohan Ding and Xiangyu Zhang and Yizhuang Zhou and Jungong Han and Guiguang Ding and Jian Sun},\n  year = {2022},\n  eprint = {2203.06717},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.06717}\n}\n","citationPath":"/content/papers/citations/reparameterization/replknet.c7cc2407aa19.bib"},"thumbnail":{"src":"/content/papers/assets/b85fa678b6e815b9d112b67d68443295a98c80478664ec8053c2e832d89574ca.jpg","width":1140,"height":972},"locale":"en","path":"/content/papers/en/reparameterization/replknet.271ac18a9d7c.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"a-guide-to-using-large-kernels-in-convolution","text":"A Guide to Using Large Kernels in Convolution","depth":2},{"id":"deep-large-kernels-remain-efficient","text":"Deep Large Kernels Remain Efficient","depth":3},{"id":"large-kernels-heavily-rely-on-residual-connections","text":"Large Kernels Heavily Rely on Residual Connections","depth":3},{"id":"large-kernels-require-reparameterization","text":"Large Kernels Require Reparameterization","depth":3},{"id":"large-kernels-significantly-enhance-downstream-task-performance","text":"Large Kernels Significantly Enhance Downstream Task Performance","depth":3},{"id":"large-kernels-are-effective-even-with-small-feature-maps","text":"Large Kernels Are Effective Even with Small Feature Maps","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"evaluation-of-large-kernels","text":"Evaluation of Large Kernels","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"large-kernel-cnns-have-larger-erf-than-deep-small-kernel-models","text":"Large Kernel CNNs Have Larger ERF Than Deep Small Kernel Models","depth":3},{"id":"shape-bias","text":"Shape Bias","depth":3},{"id":"limitations","text":"Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]},{"id":"classic-cnns/slak","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SLaK","subtitle":"Kernel Pro Max","author":"Z. Yuan","paperDate":"2022-07-07","publishedOn":"2025-03-09","world":"external/slak","paperTitle":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","paperUrl":"https://arxiv.org/abs/2207.03620","bibliography":{"title":"More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity","authors":["Shiwei Liu","Tianlong Chen","Xiaohan Chen","Xuxi Chen","Qiao Xiao","Boqian Wu","Tommi Kärkkäinen","Mykola Pechenizkiy","Decebal Mocanu","Zhangyang Wang"],"url":"https://arxiv.org/abs/2207.03620","pdfUrl":"https://arxiv.org/pdf/2207.03620v3","codeUrl":null,"preprint":{"id":"2207.03620","submittedOn":"2022-07-07","url":"https://arxiv.org/abs/2207.03620"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11529"},"citation":"@misc{classiccnnsslak2022,\n  title = {{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}},\n  author = {Shiwei Liu and Tianlong Chen and Xiaohan Chen and Xuxi Chen and Qiao Xiao and Boqian Wu and Tommi Kärkkäinen and Mykola Pechenizkiy and Decebal Mocanu and Zhangyang Wang},\n  year = {2022},\n  eprint = {2207.03620},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.03620}\n}\n","citationPath":"/content/papers/citations/classic-cnns/slak.e167e693df01.bib"},"thumbnail":{"src":"/content/papers/assets/b5b8ba279efe42831c7ec4e60395d0559da26774c8519a438783f49f4b814a85.png","width":1640,"height":660},"locale":"en","path":"/content/papers/en/classic-cnns/slak.5a1c38b2740d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"beyond-31","text":"Beyond 31","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"step-1-split-it-into-two-strips","text":"Step 1: Split It into Two Strips","depth":3},{"id":"step-2-a-little-sparser-a-little-wider","text":"Step 2: A Little Sparser, a Little Wider","depth":3},{"id":"what-slak-looks-like","text":"What SLaK Looks Like","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"running-it-for-real","text":"Running It for Real","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/resnext","relation":"referenced","topic":""}]}],"text":"In 2022, vision models with more than a billion parameters were almost all Transformers. ViT-G has 1.8 billion parameters, SwinV2-G reached 3 billion, and BEiT-3 has 1.9 billion. On the convolutional side, the largest models, ConvNeXt-XL and RepLKNet-XL, both stopped at a little over 300 million. ConvNeXt proved that convolutions can still hold their own at ordinary sizes, but at the scale of a billion parameters and hundreds of millions of images, nobody had yet built a convolutional network that could rival ViT.\n\nThis paper comes from Shanghai AI Laboratory, SenseTime, Tsinghua University, Nanjing University, and CUHK. On the author list, Jifeng Dai and Xizhou Zhu are the first authors of the first and second versions of deformable convolution, respectively, and this time they bring out their own operator to build a backbone for large models.\n\n\n> [!TIP]\n> \n> For deformable convolution, you can start with our note:\n> \n> - **[[17.03] Deformable ConvNets: Fancy Footwork](/blog/papers/classic-cnns/deformable-convnets)**\n\n\n## Defining the Problem\n\n### Where Convolutions Fall Short\n\nThe authors first compare ordinary convolution with multi-head self-attention (MHSA) and pin down two differences.\n\nThe first is long-range dependence. Stack 3×3 convolutions layer upon layer, and the effective receptive field actually stays small; however deep the network gets, it struggles to see the whole image from the start the way ViT does.\n\nThe second is adaptive spatial aggregation. Self-attention weights change with the input, differing for every image and every position. Convolution weights are fixed once training is done, and they carry strong inductive biases such as locality and translation equivariance. Inductive biases let convolutional networks converge fast and get by with less data, but they also limit how well the networks can learn more general patterns from web-scale data.\n\n[![Four-panel diagram comparing four core operators, each with a blue star in the middle for the query pixel. (a) Global attention: the whole image is covered with green cells of varying shades representing adaptive weights, marked as having long-range dependence and adaptive aggregation but not saving computation or memory. (b) Local attention: only a small patch of green cells to the upper left of the star, marked as lacking long-range dependence. (c) Large kernel: a large patch of gray cells representing fixed weights, marked as lacking adaptive aggregation. (d) The authors' dynamic sparse kernel: nine small green squares scattered around the star at varying distances, with all three items checked](/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png)](/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png)\n\n*Figure 1 from the paper, excerpted from [**page 1 of the v4 PDF**](https://arxiv.org/pdf/2211.05778v4#page=1), CC BY 4.0. Gray marks fixed weights, and green marks weights that change with the input.*\n\nLarge kernels like RepLKNet and SLaK made up for the first item, but the second is still missing, since their weights are just as fixed after training. Deformable convolution has both. Each sampling point's offset lets it look near or far, and both the offsets and the modulation weights are computed from the input.\n\n\n> [!TIP]\n> \n> For the large-kernel route, see these two notes:\n> \n> - **[[22.03] RepLKNet: Giant Convolutional Kernels](/blog/papers/reparameterization/replknet)**\n> - **[[22.07] SLaK: Kernel Pro Max](/blog/papers/classic-cnns/slak)**\n\n\n### Why DCNv2 Can't Be Used As-Is\n\nDCNv2's output at position $p_0$ can be written as\n\n$$\ny(p_0)=\\sum_{k=1}^{K} w_k\\, m_k\\, x\\!\\left(p_0+p_k+\\Delta p_k\\right),\n$$\n\nwhere $K$ is the number of sampling points, with $K=9$ for 3×3. $p_k$ is a grid position of ordinary convolution, $\\Delta p_k$ is the learned offset, $m_k$ is the modulation weight after a sigmoid, and $w_k$ is the projection weight of the $k$-th sampling point, of size $C\\times C$.\n\nThe catch is that DCNv2 is usually used as an add-on, loading weights pretrained with ordinary convolution and then switching to deformable convolution for fine-tuning. For training a billion-parameter backbone from scratch, it has two flaws. Every sampling point has its own set of $C\\times C$ weights, so parameter count and memory both grow with $K$. And the sigmoid puts each $m_k$ between 0 and 1, so the nine of them can add up to anything from 0 to 9; once the model is big and the data plentiful, gradients become unstable.\n\n## Solving the Problem\n\n### DCNv3's Three Changes\n\n**First, sampling points share weights.** Borrowing the idea of separable convolution, $w_k$ is split in two. The position-dependent part is handed to the modulation weight $m_k$, which plays the role of depthwise; the position-independent projection $w$ is shared by all sampling points, which plays the role of pointwise. Nine sets of $C\\times C$ weights shrink to one.\n\n**Second, grouping.** Following multi-head attention, the channels are split into $G$ groups, each with its own offsets and modulation weights, so different groups in the same layer can look at different places.\n\n**Third, softmax for the modulation weights.** A sigmoid is computed for each point on its own; switching to one softmax over all nine points fixes their sum at 1.\n\nPut together, this is DCNv3:\n\n$$\ny(p_0)=\\sum_{g=1}^{G}\\sum_{k=1}^{K} w_g\\, m_{gk}\\, x_g\\!\\left(p_0+p_k+\\Delta p_{gk}\\right),\n$$\n\nwhere $x_g$ is the $g$-th group of channels, each group having $C' = C/G$ channels, and $m_{gk}$ goes through a softmax along $K$.\n\nThe offsets are fractional, so sampling positions usually land between pixels, and the values have to be fetched with bilinear interpolation. Let's write the simplest version with PyTorch's `grid_sample`, leaving out the final shared projection for now:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ndef dcnv3_core(x, offset, logits, G):\n    # x: (C, H, W); offset: (G, 9, 2, H, W), in pixels; logits: (G, 9, H, W)\n    C, H, W = x.shape\n    m = logits.softmax(dim=1)                                  # each group's 9 sampling-point weights sum to 1\n    ys, xs = torch.meshgrid(torch.arange(H), torch.arange(W), indexing=\"ij\")\n    grid = [(dy, dx) for dy in (-1, 0, 1) for dx in (-1, 0, 1)]  # the 3×3 base positions p_k\n    out = torch.zeros_like(x)\n    for g, xg in enumerate(x.split(C // G)):                   # each group has its own offsets and weights\n        for k, (dy, dx) in enumerate(grid):\n            py = ys + dy + offset[g, k, 0]                     # p0 + p_k + Δp\n            px = xs + dx + offset[g, k, 1]\n            pos = torch.stack([px / (W - 1) * 2 - 1, py / (H - 1) * 2 - 1], -1)\n            v = F.grid_sample(xg[None], pos[None], align_corners=True)[0]   # fetch values by bilinear interpolation\n            out[g * (C // G):(g + 1) * (C // G)] += m[g, k] * v\n    return out                                                 # followed by a linear projection shared by all sampling points\n\ntorch.manual_seed(0)\nC, H, W, G = 8, 16, 16, 2\nx = torch.randn(C, H, W)\n\n# All offsets 0 and all weights equal: reduces to a 3×3 average\nzero_off, flat = torch.zeros(G, 9, 2, H, W), torch.zeros(G, 9, H, W)\nout = dcnv3_core(x, zero_off, flat, G)\nprint(\"With offsets at 0, max difference from 3×3 average pooling:\", f\"{(out - F.avg_pool2d(x, 3, 1, 1)).abs().max():.1e}\")\n\n# Move all 9 points of group 0 down by 5 cells: the 3×3 kernel sees 5 cells away\noff = zero_off.clone()\noff[0, :, 0] = 5.0\nout = dcnv3_core(x, off, flat, G)\nref = F.avg_pool2d(x, 3, 1, 1)\nprint(\"After moving group 0 down 5 cells, max difference from the average 5 cells below:\", f\"{(out[:4, :-5] - ref[:4, 5:]).abs().max():.1e}\")\nprint(\"Group 1 unaffected, max difference:\", f\"{(out[4:] - ref[4:]).abs().max():.1e}\")\n```\n\n```\nWith offsets at 0, max difference from 3×3 average pooling: 2.4e-07\nAfter moving group 0 down 5 cells, max difference from the average 5 cells below: 1.8e-07\nGroup 1 unaffected, max difference: 2.4e-07\n```\n\nWhen the offsets are 0 and the softmax weights are uniform, DCNv3 is just a 3×3 average. Once the offsets move, the same nine sampling points can reach 5 cells away, and only that group moves along. In the actual model, both the offsets and the modulation weights are computed from the input by a 3×3 depthwise convolution followed by a linear layer, so they differ at every position. The official code also includes a pure PyTorch version, whose comments state it's meant only for debugging and testing; training uses the CUDA operator.\n\nThe softmax item looks like a minor tweak, so let's print the sums under both approaches and compare:\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nfor scale in (0.5, 2.0, 5.0):\n    logits = torch.randn(10000, 9) * scale        # 10,000 positions, 9 sampling points each\n    s = torch.sigmoid(logits).sum(1)               # DCNv2: each point through its own sigmoid\n    t = torch.softmax(logits, 1).sum(1)            # DCNv3: the 9 points through one softmax together\n    print(f\"logit std {scale}: sigmoid sum {s.min():.2f} to {s.max():.2f}, softmax sum {t.min():.2f} to {t.max():.2f}\")\n```\n\n```\nlogit std 0.5: sigmoid sum 3.07 to 5.87, softmax sum 1.00 to 1.00\nlogit std 2.0: sigmoid sum 0.95 to 8.13, softmax sum 1.00 to 1.00\nlogit std 5.0: sigmoid sum 0.33 to 8.53, softmax sum 1.00 to 1.00\n```\n\nThe sigmoid sums can differ by more than twentyfold from one position to another, which amounts to multiplying each position's output by a different factor. The softmax sum is always 1. The ablation later on will show how much this one change matters.\n\n### Stacking It into a Network\n\nWith the operator in hand, the next question is how to build a network out of it.\n\n[![InternImage architecture diagram. On the left is the backbone: a photo of a red panda goes in, passes through the stem, then stages 1 to 4 in order, connected by downsampling layers, with the feature map shrinking from H/4×W/4 to H/32×W/32 before feeding classification, detection, segmentation, and other tasks. Three boxes on the right spell out that the stem is two stride-2 3×3 convolutions with LN and GELU, and that downsampling is a stride-2 3×3 convolution with LN; the basic block of stage i consists of DCNv3, LN, FFN, and LN with two residual connections, with the offset Δp and modulation weight m marked beside DCNv3; the four stacking rules are listed at the bottom](/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png)](/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png)\n\n*Figure 3 from the paper, excerpted from [**page 4 of the v4 PDF**](https://arxiv.org/pdf/2211.05778v4#page=4), CC BY 4.0.*\n\nThe basic block looks a lot like a Transformer's. DCNv3 takes the place of self-attention, followed by an FFN with GELU, and the outputs of both branches go through LN before being added to the residual. The stem and downsampling layers stay with the convolutional-network style; the stem is two stride-2 3×3 convolutions, and downsampling is a single stride-2 3×3 convolution.\n\nEach of the four stages has a channel count $C_i$, a group count $G_i$, and a block count $L_i$, which makes 12 hyperparameters in total, too many to tune. The authors use four rules to bring them down to 4:\n\n1. $C_i = 2^{i-1} C_1$, so the channel count doubles at each stage.\n2. $G_i = C_i / C'$, so the number of channels per group, $C'$, is fixed.\n3. $L_1 = L_2 = L_4$.\n4. $L_1 \\le L_3$, so the third stage is the deepest.\n\nThe last two give the “AABA” stacking of ConvNeXt and Swin. That leaves four numbers, $C_1$, $C'$, $L_1$, and $L_3$. The authors searched 30 combinations around 30 million parameters and picked $(64, 16, 4, 18)$ as InternImage-T.\n\nTo scale up further, the authors follow EfficientNet's compound scaling, multiplying depth by $\\alpha^{\\phi}$ and width by $\\beta^{\\phi}$ under the constraint $\\alpha\\beta^{1.99}\\approx 2$, and find experimentally that $\\alpha = 1.09$ and $\\beta = 1.36$ work best. The result is six sizes:\n\n| Model | $C_1$ | $C'$ | Blocks per stage | Parameters |\n| --- | ---: | ---: | --- | ---: |\n| InternImage-T | 64 | 16 | 4, 4, 18, 4 | 30M |\n| InternImage-S | 80 | 16 | 4, 4, 21, 4 | 50M |\n| InternImage-B | 112 | 16 | 4, 4, 21, 4 | 97M |\n| InternImage-L | 160 | 16 | 5, 5, 22, 5 | 223M |\n| InternImage-XL | 192 | 16 | 5, 5, 24, 5 | 335M |\n| InternImage-H | 320 | 32 | 6, 6, 32, 6 | 1.08B |\n\n*Table: Excerpted from Table 1 of the paper.*\n\nT through XL are sized to match ConvNeXt for easy comparison; H was built separately to reach a billion parameters.\n\n## Discussion\n\n### ImageNet\n\nFirst, the ordinary sizes. T, S, and B are trained on ImageNet-1K alone for 300 epochs, with 224×224 input:\n\n| Size | Swin | ConvNeXt | SLaK | HorNet | InternImage |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| T (about 30M) | 81.3% | 82.1% | 82.5% | 83.0% | **83.5%** |\n| S (about 50M) | 83.0% | 83.1% | 83.8% | 84.0% | **84.2%** |\n| B (about 90M) | 83.5% | 83.8% | 84.0% | 84.3% | **84.9%** |\n\n*Table: Some columns excerpted from Table 2 of the paper. Models in the same row have similar parameter counts and FLOPs.*\n\nAt the same size, InternImage is highest in every row. T is 1.4 percentage points above ConvNeXt-T, and B is 1.1 higher.\n\nNow the large models. XL is first pretrained on ImageNet-22K and reaches 88.0%. H is trained with M3I pretraining on 427 million public images from LAION-400M, YFCC-15M, and CC12M, and reaches 89.6% in the end with 640×640 input. The strongest ViTs and hybrid architectures sit at about 90.5% to 90.9%, using non-public data such as JFT, a gap of about 1 percentage point.\n\n### Detection and Segmentation\n\nThe gaps on downstream tasks are bigger than on classification. On COCO with Mask R-CNN trained for 12 epochs, InternImage-T reaches 47.2 box AP, against 42.7 for Swin-T and 44.2 for ConvNeXt-T, ahead by 4.5 and 3.0 respectively. With Cascade Mask R-CNN and 36 epochs of multi-scale training, XL gets 56.2 and ConvNeXt-XL 55.2. For ADE20K semantic segmentation with UperNet, InternImage-B gets 50.8 mIoU, against 49.1 for ConvNeXt-B, 49.9 for RepLKNet-31B, and 50.2 for SLaK-B.\n\nThe paper's most quoted number is 65.4 box AP on COCO test-dev, 1.2 above the previous best, FD-SwinV2-G, with 27% fewer parameters. How that number was reached is worth spelling out. The backbone is InternImage-H, first pretrained on 427 million images. A composite-backbone technique then doubles its parameters to 2.18 billion. The detector is DINO, trained first on Objects365 for 26 epochs and then on COCO for 12. The 62.9 mIoU on ADE20K is likewise H paired with Mask2Former, plus multi-scale testing. Every trick that could be stacked was stacked, so these numbers read more like a demonstration that this backbone can handle large-scale training.\n\n\n> [!TIP]\n> \n> For the DINO detector, see this note:\n> \n> - **[[22.03] DINO: Close, but Not Close Enough](/blog/papers/object-detection/dino-detr)**\n\n\n### Ablation: The Softmax Change\n\nRemoving DCNv3's three changes one at a time, compared on InternImage-T:\n\n| Setting | ImageNet top-1 | COCO box AP | COCO mask AP |\n| --- | ---: | ---: | ---: |\n| Full DCNv3 | 83.5% | 47.2 | 42.5 |\n| No weight sharing | 83.6% | 47.4 | 42.6 |\n| No grouping | 82.3% | 43.8 | 40.0 |\n| Sigmoid instead of softmax | **65.7%** | 38.7 | 35.6 |\n\n*Table: Excerpted from Table 6 of the paper. COCO uses Mask R-CNN trained for 12 epochs.*\n\nSharing weights barely affects accuracy, but the version without it has 66.1% more parameters. Scaled up to H, weight sharing saves 42.0% of the parameters and 84.2% of GPU memory; this change is there so the model can be made big.\n\nRemoving grouping drops ImageNet by 1.2 and COCO by 3.4, so detection takes the bigger hit.\n\nSwitching softmax back to sigmoid sends ImageNet straight down to 65.7%, a loss of nearly 18 percentage points. Here is where the sums we printed earlier show their effect.\n\nThe authors also tried larger sampling grids than 3×3. 5×5 gets 83.6%, with parameters rising from 30M to 37M; 7×7 actually drops to 82.8%, with 48M parameters. The sampling points move on their own, so 3×3 is enough.\n\n[![Four copies of the same photo of a flock of sheep, labeled stage 1 to stage 4, with a blue star fixed on a sheep on the left. In stages 1 and 2, the colored sampling points crowd around the star, with zoomed-in boxes beside them; in stages 3 and 4, the sampling points spread across the whole photo, landing on other sheep and on the grass, with different colors representing different groups](/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png)](/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png)\n\n*Figure 5 from the paper, excerpted from [**page 8 of the v4 PDF**](https://arxiv.org/pdf/2211.05778v4#page=8), CC BY 4.0. The blue star is the query point, and different colors are the sampling locations of different groups.*\n\nThe distribution of sampling points has layers to it too. In the first two stages they crowd near the query point, looking locally; by the third and fourth stages they spread across the whole image. That's very different from ViT, which is global from the first layer.\n\n### The Cost\n\nThe biggest cost is speed. In the appendix, the authors measured throughput on a single A100, with batch size 256 for 224×224 and batch size 2 for 800×800:\n\n| Model | ImageNet top-1 | 224×224 (images/s) | 800×800 (images/s) |\n| --- | ---: | ---: | ---: |\n| ConvNeXt-B | 83.8% | 881 | 58 |\n| RepLKNet-B | 83.5% | 884 | 21 |\n| DAT-B | 84.0% | 661 | 24 |\n| InternImage-B (DCNv2) | — | 311 | 16 |\n| InternImage-B | **84.9%** | 775 | 54 |\n\n*Table: Excerpted from Table 11 of the paper.*\n\nWith DCNv3, throughput is 2.5 to 3.4 times that of the DCNv2 version. It's still a bit slower than ConvNeXt-B, 12% lower at 224 and 7% lower at 800. At high resolution, though, it's 2.5 times RepLKNet-B, since large kernels struggle on big images. In their conclusion, the authors also acknowledge that the latency of DCN-style operators remains a problem for downstream tasks that demand speed.\n\nThe other thing worth a look is the amount of data. The authors uniformly sampled 1%, 10%, and all of ImageNet-1K, and trained on each for 300 epochs:\n\n| Model | 1% | 10% | 100% |\n| --- | ---: | ---: | ---: |\n| ResNet-50 | **12.2%** | **57.5%** | 80.4% |\n| ConvNeXt-T | 8.4% | 52.6% | 82.1% |\n| Swin-T | Failed | 12.1% | 81.3% |\n| InternImage-T | 5.9% | 56.0% | **83.5%** |\n\n*Table: Excerpted from Table 12 of the paper.*\n\nThe authors say InternImage-T is robust on both 1% and 10% of the data. In the 1% column, however, it loses to ResNet-50 and ConvNeXt-T and beats only Swin-T, which failed to train at all; at 10% it still loses to ResNet-50. It does cope with small amounts of data better than Swin-T. When the data shrinks to just 1%, though, ResNet-50 and ConvNeXt-T, with their stronger inductive biases, still come out ahead.\n\n## Conclusion\n\nThose record-breaking numbers really were won by brute force. It took 427 million images, a composite backbone with doubled parameters, Objects365 pretraining, and DINO on top to get COCO's 65.4.\n\nBut bring the size back down to ordinary scales, with parameters and FLOPs matched to ConvNeXt, and InternImage is still more than 1 percentage point ahead on ImageNet and 1.8 to 3.0 ahead in COCO box AP. Here there's no brute force to lean on; the credit goes to DCNv3 itself.\n\nIf you want to run it yourself, you'll first have to get on good terms with that custom CUDA operator. At every checkpoint, whether compiling, deploying, or exporting to TensorRT, it's the first thing you have to get past.\n\nOne last little aside. In this paper's ablation, switching softmax back to sigmoid makes accuracy crash from 83.5% to 65.7%. A little over a year later, much the same group released DCNv4, and one of its two main changes was dropping the softmax entirely, which even made it converge faster.\n\nThe shelf life of a paper's key design is sometimes just a year.\n","imageSizes":{"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png":{"width":1012,"height":1179},"/content/papers/assets/902dfc93f6ae81a66758189ee04f89df167af4dc79a2eee9960400c444fd2515.png":{"width":1017,"height":1467},"/content/papers/assets/0e4e2959c920bda3aad3e121de44c615afd08bade1ea30c49dbe8f20bd801598.png":{"width":1012,"height":666}}}