{"notes":[{"id":"classic-cnns/dcnv4","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"DCNv4","subtitle":"Unbounded Weights","author":"Z. Yuan","paperDate":"2024-01-11","publishedOn":"2025-12-29","paperTitle":"Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications","paperUrl":"https://arxiv.org/abs/2401.06197","bibliography":{"title":"Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications","authors":["Yuwen Xiong","Zhiqi Li","Yuntao Chen","Feng Wang","Xizhou Zhu","Jiapeng Luo","Wenhai Wang","Tong Lu","Hongsheng Li","Yu Qiao","Lewei Lu","Jie Zhou","Jifeng Dai"],"url":"https://arxiv.org/abs/2401.06197","pdfUrl":"https://arxiv.org/pdf/2401.06197v1","codeUrl":"https://github.com/OpenGVLab/DCNv4","preprint":{"id":"2401.06197","submittedOn":"2024-01-11","url":"https://arxiv.org/abs/2401.06197"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Xiong_Efficient_Deformable_ConvNets_Rethinking_Dynamic_and_Sparse_Operator_for_Vision_CVPR_2024_paper.html"},"citation":"@misc{classiccnnsdcnv42024,\n  title = {{Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications}},\n  author = {Yuwen Xiong and Zhiqi Li and Yuntao Chen and Feng Wang and Xizhou Zhu and Jiapeng Luo and Wenhai Wang and Tong Lu and Hongsheng Li and Yu Qiao and Lewei Lu and Jie Zhou and Jifeng Dai},\n  year = {2024},\n  eprint = {2401.06197},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2401.06197}\n}\n","citationPath":"/content/papers/citations/classic-cnns/dcnv4.f17f87f8ee11.bib"},"thumbnail":{"src":"/content/papers/assets/fe768f9a1b9df098fd7de1ed2c34ba0713cb83936e024d45ca26e85a2b340ff1.png","width":1020,"height":522},"locale":"en","path":"/content/papers/en/classic-cnns/dcnv4.c33fec87d9a7.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"why-is-sampling-just-9-points-still-slow","text":"Why Is Sampling Just 9 Points Still Slow?","depth":3},{"id":"softmax-was-borrowed","text":"Softmax Was Borrowed","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"1-remove-softmax","text":"1. Remove Softmax","depth":3},{"id":"2-move-less-data-on-the-gpu","text":"2. Move Less Data on the GPU","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-operator-itself","text":"The Operator Itself","depth":3},{"id":"flashinternimage","text":"FlashInternImage","depth":3},{"id":"swapping-it-into-other-networks","text":"Swapping It into Other Networks","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/internimage","relation":"referenced","topic":""},{"id":"classic-cnns/deformable-convnets","relation":"referenced","topic":""}]},{"id":"classic-cnns/internimage","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InternImage","subtitle":"Miracles by Brute Force?","author":"Z. Yuan","paperDate":"2022-11-10","publishedOn":"2025-09-04","world":"external/internimage","paperTitle":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","paperUrl":"https://arxiv.org/abs/2211.05778","bibliography":{"title":"InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions","authors":["Wenhai Wang","Jifeng Dai","Zhe Chen","Zhenhang Huang","Zhiqi Li","Xizhou Zhu","Xiaowei Hu","Tong Lu","Lewei Lu","Hongsheng Li","Xiaogang Wang","Yu Qiao"],"url":"https://arxiv.org/abs/2211.05778","pdfUrl":"https://arxiv.org/pdf/2211.05778v4","codeUrl":null,"preprint":{"id":"2211.05778","submittedOn":"2022-11-10","url":"https://arxiv.org/abs/2211.05778"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Wang_InternImage_Exploring_Large-Scale_Vision_Foundation_Models_With_Deformable_Convolutions_CVPR_2023_paper.html"},"citation":"@misc{classiccnnsinternimage2022,\n  title = {{InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions}},\n  author = {Wenhai Wang and Jifeng Dai and Zhe Chen and Zhenhang Huang and Zhiqi Li and Xizhou Zhu and Xiaowei Hu and Tong Lu and Lewei Lu and Hongsheng Li and Xiaogang Wang and Yu Qiao},\n  year = {2022},\n  eprint = {2211.05778},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2211.05778}\n}\n","citationPath":"/content/papers/citations/classic-cnns/internimage.4baca0899bc9.bib"},"thumbnail":{"src":"/content/papers/assets/eb597a4ac9a7c2e5e9b05e8ff1c61a951adab71d5cc5a273796762d353a9a0e9.png","width":1012,"height":1179},"locale":"en","path":"/content/papers/en/classic-cnns/internimage.02b85ef97de6.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"where-convolutions-fall-short","text":"Where Convolutions Fall Short","depth":3},{"id":"why-dcnv2-cant-be-used-as-is","text":"Why DCNv2 Can't Be Used As-Is","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"dcnv3s-three-changes","text":"DCNv3's Three Changes","depth":3},{"id":"stacking-it-into-a-network","text":"Stacking It into a Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"detection-and-segmentation","text":"Detection and Segmentation","depth":3},{"id":"ablation-the-softmax-change","text":"Ablation: The Softmax Change","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/deformable-convnets","relation":"referenced","topic":""},{"id":"reparameterization/replknet","relation":"referenced","topic":""},{"id":"classic-cnns/slak","relation":"referenced","topic":""}]},{"id":"classic-cnns/deformable-convnets","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deformable ConvNets","subtitle":"Fancy Footwork","author":"Z. Yuan","paperDate":"2017-03-17","publishedOn":"2024-05-01","world":"external/deformable-convnets","paperTitle":"Deformable Convolutional Networks","paperUrl":"https://arxiv.org/abs/1703.06211","bibliography":{"title":"Deformable Convolutional Networks","authors":["Jifeng Dai","Haozhi Qi","Yuwen Xiong","Yi Li","Guodong Zhang","Han Hu","Yichen Wei"],"url":"https://arxiv.org/abs/1703.06211","pdfUrl":"https://arxiv.org/pdf/1703.06211v3","codeUrl":null,"preprint":{"id":"1703.06211","submittedOn":"2017-03-17","url":"https://arxiv.org/abs/1703.06211"},"publication":{"venue":"ICCV","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_iccv_2017/html/Dai_Deformable_Convolutional_Networks_ICCV_2017_paper.html"},"citation":"@misc{classiccnnsdeformableconvnets2017,\n  title = {{Deformable Convolutional Networks}},\n  author = {Jifeng Dai and Haozhi Qi and Yuwen Xiong and Yi Li and Guodong Zhang and Han Hu and Yichen Wei},\n  year = {2017},\n  eprint = {1703.06211},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1703.06211}\n}\n","citationPath":"/content/papers/citations/classic-cnns/deformable-convnets.f724f3ff1784.bib"},"thumbnail":{"src":"/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png","width":987,"height":633},"locale":"en","path":"/content/papers/en/classic-cnns/deformable-convnets.319cd37ba4c1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-fixed-grid","text":"The Fixed Grid","depth":3},{"id":"someone-already-taught-networks-to-warp","text":"Someone Already Taught Networks to Warp","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-two-steps-of-convolution","text":"The Two Steps of Convolution","depth":3},{"id":"an-offset-for-every-point","text":"An Offset for Every Point","depth":3},{"id":"sampling-at-fractional-positions","text":"Sampling at Fractional Positions","depth":3},{"id":"where-the-offsets-come-from","text":"Where the Offsets Come From","depth":3},{"id":"two-special-cases","text":"Two Special Cases","depth":3},{"id":"deformable-roi-pooling","text":"Deformable RoI Pooling","depth":3},{"id":"into-the-network","text":"Into the Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"receptive-fields-really-do-follow-the-object","text":"Receptive Fields Really Do Follow the Object","depth":3},{"id":"how-many-layers-to-replace","text":"How Many Layers to Replace","depth":3},{"id":"compared-with-atrous-convolution","text":"Compared with Atrous Convolution","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"coco","text":"COCO","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/stn","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""}]}],"text":"In InternImage's ablations, replacing softmax with sigmoid for DCNv3's modulation weights drops ImageNet accuracy from 83.5% to 65.7%. Softmax looks indispensable to DCNv3.\n\nA little over a year later, largely the same group introduces DCNv4. Its first change is to remove softmax, and it converges faster.\n\nFirst author Yuwen Xiong coauthored the original deformable convolution in 2017; corresponding author Jifeng Dai was that paper's first author. This team includes the University of Toronto, Shanghai AI Laboratory, Tsinghua University, SenseTime, and the Chinese University of Hong Kong. Beyond softmax, the other half of the paper asks why deformable convolution runs slower than 7×7 depthwise convolution when it samples only 9 points.\n\n\n> [!TIP]\n> \n> For DCNv3, InternImage, and the original deformable convolution, see:\n> \n> - **[[22.11] InternImage: Miracles by Brute Force?](/blog/papers/classic-cnns/internimage)**\n> - **[[17.03] Deformable ConvNets: Fancy Footwork](/blog/papers/classic-cnns/deformable-convnets)**\n\n\n[![Figure 1 from the paper: (a) a bar chart with input tensor sizes 56×56×128, 28×28×256, and 14×14×512 on the horizontal axis, and execution time relative to DCNv3 = 1 on the vertical axis. Each group has bars for FlashAttention, Window Attention, DWConv, DCNv3, and DCNv4. DCNv4 is shortest in the first two groups and close to FlashAttention in the third; the 56×56×128 group is marked 3.76× faster than DCNv3. (b) a line chart with training iterations from 0 to 15K horizontally and ImageNet top-1 accuracy vertically. DCNv4 leads early, DWConv and Dense Attention lie in the middle, and DCNv3 starts lowest.](/content/papers/assets/5a7b5fe22fc3c1bf41919d57fbcca8c73c4d14c00875c73ce999527f2f9ae54e.png)](/content/papers/assets/5a7b5fe22fc3c1bf41919d57fbcca8c73c4d14c00875c73ce999527f2f9ae54e.png)\n\n*Figure 1 from Xiong et al., taken from [**PDF page 1**](https://arxiv.org/pdf/2401.06197v1#page=1). The original caption is omitted.*\n\nThe left plot shows speed, with DCNv3 normalized to 1 and shorter bars indicating faster execution. DCNv4 is fastest at the first two sizes and close to FlashAttention at 14×14×512. The right plot swaps operators in the same network architecture and compares early training accuracy. DCNv4 leads throughout, while DCNv3 starts at the bottom.\n\n## Defining the Problem\n\n### Why Is Sampling Just 9 Points Still Slow?\n\nFirst, a recap of DCNv3. At location $p_0$, the output for channel group $g$ is\n\n$$\ny_g(p_0)=\\sum_{k=1}^{K} m_{gk}\\, x_g\\!\\left(p_0+p_k+\\Delta p_{gk}\\right),\n$$\n\n$K=9$ is the number of sampling points on a 3×3 grid, $\\Delta p_{gk}$ is the learned offset, and $m_{gk}$ is the modulation weight, normalized by softmax along $K$ in DCNv3. Both offsets and weights are computed from the input and vary by location.\n\nA sparse operator looking at only 9 points ought to be faster than 7×7 depthwise convolution or dense attention. In practice, memory access gets in the way.\n\nThe authors analyze this with the roofline model. For input and output of size $H\\times W\\times C$, DCNv3 takes $36HWC$ FLOPs: each of 9 sampling points needs bilinear interpolation involving 4 pixels. Ideally, each value is read from memory once, for roughly $3.7HWC$ memory accesses.\n\nWith parallel computation, however, many threads read simultaneously, and cache hits are not guaranteed. In the worst case, each output location rereads 36 interpolation pixels, 27 offsets and weights—2 offsets and 1 weight for each of 9 points—and writes 1 result. Memory access grows to $64HWC$, 17 times the ideal case.\n\nThe ratio of computation to memory access is only 0.6 to 9.7, on the memory-bandwidth-bound side of the roofline model. More GPU arithmetic capacity cannot help much.\n\n### Softmax Was Borrowed\n\nThe other issue is softmax. DCNv3 borrows it from attention, but attention has its own reason for needing it.\n\nScaled dot-product attention is $\\mathrm{softmax}(QK^\\top/\\sqrt{d})V$. Without softmax, one can compute $K^\\top V$ first, obtaining a $d\\times d$ matrix. Every query in the same window then multiplies the same matrix, reducing attention to a linear projection.\n\nConvolution and DCNv3 avoid this issue. Each location already has its own sampling window with its own values. There is no “key” concept requiring this normalization.\n\nTo test this, the authors instead add softmax to ConvNeXt's 7×7 depthwise convolution weights:\n\n| Model | Epoch 5 | Epoch 10 | Epoch 20 | Epoch 50 | Epoch 300 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| ConvNeXt | 29.9% | 53.5% | 66.1% | 74.8% | 83.8% |\n| ConvNeXt + softmax | 8.5% | 25.3% | 51.1% | 69.1% | 81.6% |\n\n*Table: from the paper's Table 1, ImageNet-1K top-1.*\n\nSoftmax clearly slows convergence and costs 2.2 percentage points by the end of training.\n\nSoftmax bounds each weight between 0 and 1 and makes their sum 1, restricting the output to a weighted average of sampled values. A weighted average stays between their minimum and maximum, so it cannot perform subtraction—the very operation used by basic filters such as edge detectors. Let's have the two kinds of weights learn a “right minus left” filter:\n\n```python\nimport torch\n\ntorch.manual_seed(0)\nx = torch.randn(1000, 3)                     # One thousand locations, each sampling left, center, and right\ntarget = x[:, 2] - x[:, 0]                   # Learn right minus left, a basic edge detector\n\nfor name, to_weight in [(\"softmax weights\", lambda z: z.softmax(0)), (\"unbounded weights\", lambda z: z)]:\n    z = torch.zeros(3, requires_grad=True)\n    opt = torch.optim.Adam([z], lr=0.1)\n    for _ in range(500):\n        loss = ((x @ to_weight(z) - target) ** 2).mean()\n        opt.zero_grad()\n        loss.backward()\n        opt.step()\n    w = to_weight(z).detach()\n    print(f\"{name}: left {w[0]:+.2f}  center {w[1]:+.2f}  right {w[2]:+.2f}  error {loss.item():.3f}\")\nprint(f\"Target variance: {target.var():.3f}\")\n```\n\n```text\nsoftmax weights: left +0.00  center +0.00  right +1.00  error 0.995\nunbounded weights: left -1.00  center +0.00  right +1.00  error 0.000\nTarget variance: 2.008\n```\n\nUnbounded weights directly learn $(-1, 0, 1)$. Softmax cannot produce negative weights, so its best option puts all the weight on the right point, leaving about half the error.\n\n[![Figure 2 from the paper: four panels compare spatial aggregation for query pixels at different locations in one channel. The upper-left legend uses a star for the query pixel and squares for response pixels; the color bar has two ranges, (0, 1) and (−∞, +∞). (a) Attention: Shared/Fixed windows, Dynamic weights, Bounded range. (b) DCNv3: Dedicated/Adaptive windows, Dynamic weights, Bounded range. (c) Convolution: Dedicated/Fixed windows, Static weights, Unbounded range. (d) DCNv4: Dedicated/Adaptive windows, Dynamic weights, Unbounded range.](/content/papers/assets/fe768f9a1b9df098fd7de1ed2c34ba0713cb83936e024d45ca26e85a2b340ff1.png)](/content/papers/assets/fe768f9a1b9df098fd7de1ed2c34ba0713cb83936e024d45ca26e85a2b340ff1.png)\n\n*Figure 2 from Xiong et al., taken from [**PDF page 3**](https://arxiv.org/pdf/2401.06197v1#page=3). The original caption is omitted.*\n\nThis figure compares the four operators. Attention has varying but bounded weights. Convolution has unbounded but fixed weights and fixed window shapes. DCNv4 targets the lower-right panel: a deformable window dedicated to each location, with input-dependent, unbounded weights.\n\n## Solving the Problem\n\n### 1. Remove Softmax\n\nThe formula stays the same as DCNv3, except that $m_{gk}$ bypasses softmax. A linear layer predicts it directly from the input, allowing any real value.\n\nRemoving softmax has a side benefit. Offsets and weights originally use two linear layers; now these can merge into one, saving a kernel launch and synchronization. The authors also simplify the offset subnetwork. It originally uses depthwise 3×3, LayerNorm, GELU, and a linear layer; removing the middle LayerNorm and GELU returns it to a simple separable convolution like Xception's.\n\n### 2. Move Less Data on the GPU\n\n[![Figure 3 from the paper: two diagrams compare thread assignments in DCNv3 and DCNv4. Each has a set of offsets and weights in GPU memory on the left and a cube-shaped H, W, C output tensor on the right. (a) DCNv3 uses Thread-1 through Thread-6, one per channel, each making a memory-read request. (b) DCNv4 uses only Thread-1 through Thread-3, with each handling two channels in the same group, halving read requests; the upper-right label marks a 2× reduction in memory access.](/content/papers/assets/db5cc5d4b75082f8705ba96857cd9bf56bd46cfb53a7946d53500fabcace0dd5.png)](/content/papers/assets/db5cc5d4b75082f8705ba96857cd9bf56bd46cfb53a7946d53500fabcace0dd5.png)\n\n*Figure 3 from Xiong et al., taken from [**PDF page 4**](https://arxiv.org/pdf/2401.06197v1#page=4). The original caption is omitted.*\n\nDCNv3's original CUDA implementation launches $H\\times W\\times C$ threads, one per channel at each location. Yet the $D=C/G$ channels in a group share the same offsets and weights at a given location. All $D$ threads read the same data separately and separately compute the bilinear interpolation coefficients.\n\nDCNv4 lets each thread handle several channels within one group, reading offsets and weights once and computing interpolation coefficients once. The illustration uses two channels per thread for clarity; the actual implementation uses 8.\n\nInterestingly, this change alone makes it slower. Fewer threads reduce parallelism, and each thread has 8 times the work. The real speedup comes next. Channels are contiguous in memory, in channel-last layout, so one vectorized instruction can read adjacent channels together—for example, four 32-bit floats in one 128-bit read. Writes work the same way.\n\nThe authors measure each step separately. The input is 56×56×128, batch size 64, with 4 groups of 32 channels. Times are in milliseconds:\n\n| Implementation | Whole module | CUDA kernel |\n| --- | ---: | ---: |\n| Original DCNv3 | 3.28 | 1.45 |\n| Simplified module design | 2.12 | 1.45 |\n| One thread per 8 channels | 2.20 | 1.53 |\n| Reuse interpolation coefficients | 2.18 | 1.51 |\n| Vectorized reads and writes | 1.28 | 0.606 |\n| Half precision | **0.873** | **0.404** |\n\n*Table: from the paper's Table 10. A100.*\n\nSimplifying the module affects only work outside the CUDA kernel, leaving kernel time unchanged. The next two steps make the kernel slightly slower. Vectorized reads and writes then cut its time by 60%. Half precision uses 2 bytes per value, moving twice as many values at the same bandwidth. The authors say that half precision brings no speedup to the original DCNv3; only the new implementation benefits.\n\n## Discussion\n\n### The Operator Itself\n\nFirst, compare operators at input size 56×56×128, measured in FP32 and FP16:\n\n| Operator | FP32 (ms) | FP16 (ms) |\n| --- | ---: | ---: |\n| Global attention (PyTorch) | 30.8 | 19.3 |\n| FlashAttention-2 | — | 2.46 |\n| 7×7 window attention | 4.05 | 1.46 |\n| 7×7 DWConv (PyTorch) | 2.02 | 1.98 |\n| 7×7 DWConv (cuDNN) | 0.981 | 0.438 |\n| DCNv3 | 1.45 | 1.52 |\n| DCNv4 | **0.606** | **0.404** |\n\n*Table: from the first input-size column of the paper's Table 2. FlashAttention-2 has no FP32 version.*\n\nDCNv4 is faster even than cuDNN's 7×7 depthwise convolution, cutting FP32 execution time by nearly 40%. The DCNv3 row also confirms the earlier point: switching to FP16 makes it slightly slower.\n\n### FlashInternImage\n\nReplace DCNv3 in InternImage with DCNv4 and leave everything else unchanged; the authors call the result FlashInternImage. Throughput is measured in both FP32 and FP16:\n\n| Model | top-1 | Throughput (FP32 / FP16, images/s) |\n| --- | ---: | ---: |\n| ConvNeXt-T | 82.1% | 2485 / 4305 |\n| InternImage-T | 83.5% | 1409 / 1746 |\n| FlashInternImage-T | **83.6%** | 2316 / 3154 |\n| ConvNeXt-S | 83.1% | 1645 / 2538 |\n| InternImage-S | 84.2% | 1044 / 1321 |\n| FlashInternImage-S | **84.4%** | 1625 / 2396 |\n| ConvNeXt-B | 83.8% | 1241 / 1888 |\n| InternImage-B | 84.9% | 779 / 1030 |\n| FlashInternImage-B | **84.9%** | 1174 / 1816 |\n\n*Table: selected rows from the paper's Table 4.*\n\nAgainst InternImage, T is 64% to 80% faster, with slightly better accuracy.\n\nThe ConvNeXt comparison needs a closer look. In FP16, FlashInternImage-T still runs at only a little over 70% of ConvNeXt-T's speed. S and B are also slightly slower, at roughly 95%. A fair description is that DCNv4 brings InternImage close to ConvNeXt speed, with 1.1 to 1.5 percentage points higher accuracy.\n\nDownstream tasks follow the same trend. On COCO with Mask R-CNN trained for 12 epochs, FlashInternImage-T gets 48.0 box AP against InternImage-T's 47.2, and FPS rises from 54 / 69 to 72 / 102. On ADE20K with UperNet, single-scale mIoU rises from 47.9 to 49.3.\n\n### Swapping It into Other Networks\n\nThe authors also use DCNv4 as a drop-in component in other architectures, keeping the other settings and hyperparameters unchanged:\n\n- **ConvNeXt-B**: replacing 7×7 depthwise convolution with DCNv4 raises ImageNet from 83.8% to 84.0%, with 20% more throughput in FP32 and 33% in FP16.\n- **ViT-B**: replacing attention with DCNv4 raises 81.8% to 81.9%, with 24% and 17% more throughput than the original using FlashAttention-2.\n- **Latent diffusion U-Net**: replacing attention modules and standard 3×3 convolutions lowers FID from 2.94 to 2.44 on ImageNet 256×256 class-conditional generation, with parameters dropping from 860M to 566M.\n\nThese architectures were tuned for their own operators, yet swapping DCNv4 straight in preserves or slightly improves accuracy while increasing speed. For generation, the authors also say the architecture and hyperparameters may need further tuning for DCNv4.\n\n### The Cost\n\nDCNv4 remains a custom CUDA operator. Many issues in the official repository concern build failures, missing modules, lack of CPU support, and difficulties exporting to ONNX or TensorRT. The repository's last push was in May 2024.\n\nSome users also report results differing from the paper. One finds [training slower than DCNv3](https://github.com/OpenGVLab/DCNv4/issues/54) on their segmentation task. Another uses the official speed script on a 3090 and [measures slower results than reported](https://github.com/OpenGVLab/DCNv4/issues/106). Several users integrating DCNv4 into their own models, such as YOLO, report [loss failing to decrease](https://github.com/OpenGVLab/DCNv4/issues/80) or [NaNs](https://github.com/OpenGVLab/DCNv4/issues/105). These reports lack full configurations and have no official replies, so the causes remain unknown.\n\nThis note's hypothesis is that removing softmax's bound may require retuning initialization and learning rate when integrating the operator into another network. The paper's successful results come from architectures built for deformable convolution, such as InternImage, or settings tuned by the authors.\n\n## Conclusion\n\nDCNv4 does two things. It removes softmax, leaving weights unbounded as in ordinary convolution and speeding convergence. It also rewrites the CUDA kernel: channels in the same group share one read, and vectorized instructions move several channels at a time, making the operator more than three times as fast.\n\nIf you already use InternImage, FlashInternImage brings speed at almost no extra cost. For your own model, enable half precision—DCNv4 is fastest in FP16—and keep an eye on loss during the first few hundred steps.\n\nThe title says “rethinking dynamic and sparse operators.” By the end, much of the speed comes from a plain question: how many times should the same group of channels read the same offsets from memory?\n","imageSizes":{"/content/papers/assets/5a7b5fe22fc3c1bf41919d57fbcca8c73c4d14c00875c73ce999527f2f9ae54e.png":{"width":1020,"height":546},"/content/papers/assets/fe768f9a1b9df098fd7de1ed2c34ba0713cb83936e024d45ca26e85a2b340ff1.png":{"width":1020,"height":522},"/content/papers/assets/db5cc5d4b75082f8705ba96857cd9bf56bd46cfb53a7946d53500fabcace0dd5.png":{"width":1020,"height":426}}}