{"notes":[{"id":"classic-cnns/rdnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RDNet","subtitle":"Don't Add, Concatenate","author":"Z. Yuan","paperDate":"2024-03-28","publishedOn":"2026-01-14","world":"external/rdnet","paperTitle":"DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs","paperUrl":"https://arxiv.org/abs/2403.19588","bibliography":{"title":"DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs","authors":["Donghyun Kim","Byeongho Heo","Dongyoon Han"],"url":"https://arxiv.org/abs/2403.19588","pdfUrl":"https://arxiv.org/pdf/2403.19588v2","codeUrl":"https://github.com/naver-ai/rdnet","preprint":{"id":"2403.19588","submittedOn":"2024-03-28","url":"https://arxiv.org/abs/2403.19588"},"publication":{"venue":"ECCV","year":2024,"kind":"conference","url":"https://doi.org/10.1007/978-3-031-72646-0_23"},"citation":"@misc{classiccnnsrdnet2024,\n  title = {{DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs}},\n  author = {Donghyun Kim and Byeongho Heo and Dongyoon Han},\n  year = {2024},\n  eprint = {2403.19588},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2403.19588}\n}\n","citationPath":"/content/papers/citations/classic-cnns/rdnet.0ac4484543c0.bib"},"thumbnail":{"src":"/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png","width":1230,"height":615},"locale":"en","path":"/content/papers/en/classic-cnns/rdnet.443c4dc8f53f.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"addition-versus-concatenation","text":"Addition versus Concatenation","depth":3},{"id":"first-sample-fifteen-thousand-random-networks","text":"First, Sample Fifteen Thousand Random Networks","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"starting-from-densenet-201","text":"Starting from DenseNet-201","depth":3},{"id":"following-the-channel-counts","text":"Following the Channel Counts","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"stochastic-depth-helps-too","text":"Stochastic Depth Helps Too","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/densenet","relation":"referenced","topic":""},{"id":"classic-cnns/cspnet","relation":"referenced","topic":""},{"id":"classic-cnns/regnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/densenet","category":"classic-cnns","legacyPath":"classic-cnns/1608-densenet","legacySlug":"1608-densenet","legacyPaths":[],"title":"DenseNet","subtitle":"Connecting Everything","author":"Z. Yuan","paperDate":"2016-08-25","publishedOn":"2024-07-02","paperTitle":"Densely Connected Convolutional Networks","paperUrl":"https://arxiv.org/abs/1608.06993","bibliography":{"title":"Densely Connected Convolutional Networks","authors":["Gao Huang","Zhuang Liu","Laurens van der Maaten","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1608.06993","pdfUrl":"https://arxiv.org/pdf/1608.06993","codeUrl":null,"preprint":{"id":"1608.06993","submittedOn":"2016-08-25","url":"https://arxiv.org/abs/1608.06993"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1608.06993"},"citation":"@misc{classiccnnsdensenet2016,\n  title = {{Densely Connected Convolutional Networks}},\n  author = {Gao Huang and Zhuang Liu and Laurens van der Maaten and Kilian Q. Weinberger},\n  year = {2016},\n  eprint = {1608.06993},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1608.06993}\n}\n","citationPath":"/content/papers/citations/classic-cnns/densenet.6f28153b67f7.bib"},"thumbnail":{"src":"/content/papers/assets/4f073a0fd1fc5b7415ca19c48934d180f9043fddc08301819a50ceb1b0ebfc69.jpg","width":1184,"height":828},"locale":"en","path":"/content/papers/en/classic-cnns/densenet.9b1c0f86199d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"transition-layers-and-average-pooling","text":"Transition Layers and Average Pooling","depth":3},{"id":"growth-rate","text":"Growth Rate","depth":3},{"id":"bottleneck-layers","text":"Bottleneck Layers","depth":3},{"id":"compression","text":"Compression","depth":3},{"id":"implementation-details","text":"Implementation Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-results","text":"Experimental Results","depth":3},{"id":"compact-models","text":"Compact Models","depth":3},{"id":"feature-reuse","text":"Feature Reuse","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"background","topic":"Training deeper convolutional networks"}]},{"id":"classic-cnns/cspnet","category":"classic-cnns","legacyPath":"classic-cnns/1911-cspnet","legacySlug":"1911-cspnet","legacyPaths":[],"title":"CSPNet","subtitle":"Half-Half-Half Plugin","author":"Z. Yuan","paperDate":"2019-11-27","publishedOn":"2024-09-29","paperTitle":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","paperUrl":"https://arxiv.org/abs/1911.11929","bibliography":{"title":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","authors":["Chien-Yao Wang","Hong-Yuan Mark Liao","I-Hau Yeh","Yueh-Hua Wu","Ping-Yang Chen","Jun-Wei Hsieh"],"url":"https://arxiv.org/abs/1911.11929","pdfUrl":"https://arxiv.org/pdf/1911.11929","codeUrl":null,"preprint":{"id":"1911.11929","submittedOn":"2019-11-27","url":"https://arxiv.org/abs/1911.11929"},"publication":{"venue":"CVPR Workshops","year":2020,"kind":"workshop","url":"https://doi.org/10.1109/cvprw50498.2020.00203"},"citation":"@misc{classiccnnscspnet2019,\n  title = {{CSPNet: A New Backbone that can Enhance Learning Capability of CNN}},\n  author = {Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},\n  year = {2019},\n  eprint = {1911.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1911.11929}\n}\n","citationPath":"/content/papers/citations/classic-cnns/cspnet.2914b9492a0c.bib"},"thumbnail":{"src":"/content/papers/assets/2602af8b5f92cba820e545c84abd94943cab1fe2631cf83834972b54b374ed20.jpg","width":2809,"height":1911},"locale":"en","path":"/content/papers/en/classic-cnns/cspnet.9c6ad42be111.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"transition-layer","text":"Transition Layer","depth":3},{"id":"exact-fusion-model","text":"Exact Fusion Model","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"effectiveness-of-the-csp-architecture","text":"Effectiveness of the CSP Architecture","depth":3},{"id":"effectiveness-of-the-efm-architecture","text":"Effectiveness of the EFM Architecture","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"performance-on-ms-coco","text":"Performance on MS COCO","depth":3},{"id":"inference-speed","text":"Inference Speed","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"},{"id":"classic-cnns/resnext","relation":"comparison","topic":"Branches, grouped convolutions and feature reuse"}]},{"id":"classic-cnns/regnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RegNet","subtitle":"Do Good Architectures All Look Alike?","author":"Z. Yuan","paperDate":"2020-03-30","publishedOn":"2024-12-16","world":"external/regnet","paperTitle":"Designing Network Design Spaces","paperUrl":"https://arxiv.org/abs/2003.13678","bibliography":{"title":"Designing Network Design Spaces","authors":["Ilija Radosavovic","Raj Prateek Kosaraju","Ross Girshick","Kaiming He","Piotr Dollár"],"url":"https://arxiv.org/abs/2003.13678","pdfUrl":"https://arxiv.org/pdf/2003.13678v1","codeUrl":null,"preprint":{"id":"2003.13678","submittedOn":"2020-03-30","url":"https://arxiv.org/abs/2003.13678"},"publication":{"venue":"CVPR","year":2020,"kind":"conference","url":"https://openaccess.thecvf.com/content_CVPR_2020/html/Radosavovic_Designing_Network_Design_Spaces_CVPR_2020_paper.html"},"citation":"@misc{classiccnnsregnet2020,\n  title = {{Designing Network Design Spaces}},\n  author = {Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Dollár},\n  year = {2020},\n  eprint = {2003.13678},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2003.13678}\n}\n","citationPath":"/content/papers/citations/classic-cnns/regnet.51d4a967c43d.bib"},"thumbnail":{"src":"/content/papers/assets/477f84998189b88eda3ae1c093bfad949ef34cf53e00368019c348b971c3a63d.png","width":1013,"height":550},"locale":"en","path":"/content/papers/en/classic-cnns/regnet.5bb32cbae384.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"what-manual-design-and-nas-each-lack","text":"What Manual Design and NAS Each Lack","depth":3},{"id":"how-to-compare-two-design-spaces","text":"How to Compare Two Design Spaces","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-starting-point-anynet","text":"The Starting Point: AnyNet","depth":3},{"id":"narrowing-it-down-step-by-step","text":"Narrowing It Down Step by Step","depth":3},{"id":"width-is-a-straight-line","text":"Width Is a Straight Line","depth":3},{"id":"regnet","text":"RegNet","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"where-it-breaks-with-common-practice","text":"Where It Breaks with Common Practice","depth":3},{"id":"compared-with-resnet-and-resnext","text":"Compared with ResNet and ResNeXt","depth":3},{"id":"the-mobile-regime","text":"The Mobile Regime","depth":3},{"id":"compared-with-efficientnet","text":"Compared with EfficientNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/nasnet","relation":"referenced","topic":""},{"id":"classic-cnns/efficientnet","relation":"referenced","topic":""}]}],"text":"There are two ways to make a shortcut connection. ResNet adds input to output, $x + f(x)$; DenseNet concatenates them, $[x, f(x)]$.\n\nDenseNet matched ResNet with fewer parameters in 2016 and won the CVPR 2017 Best Paper Award. Yet the mainstream networks that followed—ResNeXt, EfficientNet, ConvNeXt, then ViT and Swin—almost all used addition. DenseNet gradually faded from discussions of mainstream architectures.\n\nThis paper comes from NAVER Cloud AI and NAVER AI Lab, by Donghyun Kim, Byeongho Heo, and Dongyoon Han. They argue that DenseNet's neglect stems from outdated training and design details. They keep concatenation and modernize everything else to see how far DenseNet can go.\n\n\n> [!TIP]\n> \n> For the original DenseNet design and CSPNet, which later borrowed some of its ideas, see:\n> \n> - **[[16.08] DenseNet: Connecting Everything](/blog/papers/classic-cnns/densenet)**\n> - **[[19.11] CSPNet: Half-Half-Half Plugin](/blog/papers/classic-cnns/cspnet)**\n\n\n## Defining the Problem\n\n### Addition versus Concatenation\n\nAn additive shortcut keeps output width fixed. However many layers you stack, features remain $d$ channels wide; each layer can only change the contents of those $d$ channels.\n\nA concatenative shortcut adds $g$ channels at every layer. Here $g$ is the growth rate, or GR. All preceding features remain unchanged and directly available to later layers.\n\nThe authors explain the benefit through matrix rank. For a layer output $f(XW)$, $X$ is an $N\\times d_{in}$ input and $W$ is a $d_{in}\\times d_{out}$ weight matrix. With nonlinearity, output rank generally approaches $d_{out}$. Higher output dimensionality brings greater expressive capacity. Concatenation is equivalent to writing $W=[W_P, I]$: retain the original $d_{in}$ channels and add $g$ channels produced by $W_P$, so rank keeps growing.\n\nThat is also DenseNet's old problem. More channels keep increasing memory use, so the original uses a small GR, such as 32, and compensates with great depth. DenseNet-201 has four dense blocks of 6, 12, 48, and 32 layers. Deep, narrow networks run slowly on GPUs.\n\nThe authors further argue that transition layers can compress channels without losing much rank, provided the compressed dimension remains large enough. Frequent compression can therefore control memory while preserving much of the expressive capacity.\n\n### First, Sample Fifteen Thousand Random Networks\n\nThese are still hypotheses, so the authors test them first. On Tiny-ImageNet, they randomly generate over 15,000 networks, divided into additive and concatenative shortcut groups. Depth, width, activation, normalization, and kernel size are randomized too. Only networks within the compute budget are trained, giving the two shortcut types similar computational costs.\n\nThere are five random search spaces. A, B, and C have three different sizes; D adds augmentation to C, and E further switches the optimizer to AdamW:\n\n| Space | Addition top-1 | Concatenation top-1 | Addition memory (GB) | Concatenation memory (GB) |\n| --- | ---: | ---: | ---: | ---: |\n| A (about 2.2M parameters) | 45.8 ± 2.0% | **47.5 ± 2.1%** | 0.65 | 0.75 |\n| B (about 4.5M) | 50.1 ± 2.1% | **51.2 ± 2.1%** | 0.78 | 0.90 |\n| C (about 9.5M) | 53.2 ± 2.2% | **54.3 ± 2.0%** | 1.01 | 1.24 |\n| D (C with augmentation) | 57.4 ± 1.5% | **58.1 ± 1.4%** | 1.02 | 1.24 |\n| E (D with AdamW) | 58.2 ± 1.5% | **58.9 ± 1.6%** | 1.02 | 1.25 |\n\n*Table: from the paper's Table 7; mean and standard deviation over all networks in each space.*\n\nConcatenation has higher mean accuracy in all five spaces, by 0.7 to 1.7 percentage points. Standard deviations are around 2 points, though, so the distributions overlap heavily. Stronger training recipes also narrow the gap from 1.1 to 0.7. Meanwhile, concatenation uses 15% to 23% more memory; DenseNet's old problem persists.\n\nThese experiments use small 64×64 images and networks under 10 million parameters. They indicate a direction, with the behavior of larger networks still to be checked. Next comes an actual ImageNet model.\n\n\n> [!TIP]\n> \n> The approach of sampling many networks and comparing accuracy distributions comes from RegNet's authors. See:\n> \n> - **[[20.03] RegNet: Do Good Architectures All Look Alike?](/blog/papers/classic-cnns/regnet)**\n\n\n## Solving the Problem\n\n### Starting from DenseNet-201\n\nThe approach resembles ConvNeXt's: start with an old model, change one thing at a time, and record accuracy and speed at each step. DenseNet-201 with a modern recipe—AdamW, cosine learning rate, 300 epochs, and assorted augmentations—starts at 79.7% ImageNet accuracy.\n\n| Step | top-1 | Parameters | FLOPs | Inference latency (batch 1) | Training latency | Training memory |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| DenseNet-201 (new recipe) | 79.7% | 20.0M | 4.3G | 38.4 ms | 131 ms | 3.9 GB |\n| Wider and shallower | 79.5% | 21.8M | 11.1G | 8.5 ms | 85 ms | 3.2 GB |\n| Modern blocks | 80.4% | 12.9M | 4.8G | 10.4 ms | 112 ms | 3.4 GB |\n| Expansion relative to input | 80.8% | 19.9M | 4.7G | 11.8 ms | 88 ms | 3.1 GB |\n| More transitions, stage-specific GR | 82.3% | 21.2M | 5.0G | 11.0 ms | 90 ms | 3.4 GB |\n| Patchify stem | 82.4% | 21.2M | 4.9G | 11.0 ms | 88 ms | 3.2 GB |\n| Redesigned transitions | 82.6% | 22.4M | 4.9G | 13.6 ms | 97 ms | 3.1 GB |\n| Channel rescaling | **82.8%** | 23.9M | 5.0G | 14.0 ms | 99 ms | 3.1 GB |\n\n*Table: selected columns from the paper's Table 1.*\n\nTake these step by step.\n\n**Wider and shallower.** GR rises from 32 to 120, while block counts across the four stages shrink from $(6, 12, 48, 32)$ to $(3, 3, 12, 3)$. Accuracy barely changes; training latency falls 35% and memory 18%. FLOPs rise to 11.1G, to be addressed later.\n\n**Modern blocks.** Each block takes a ConvNeXt-like form: 7×7 depthwise convolution, LayerNorm, then 1×1 convolutions expanding and compressing channels, with fewer normalizations and activations. The difference is that output width equals GR and is smaller than input width. Every block compresses all the preceding concatenated features into GR new channels. This adds 0.9 percentage points.\n\n**Expansion relative to input.** The original DenseNet bottleneck has width 4 times GR. The authors change it to 4 times the input channel count, enriching the features before the nonlinearity. They then halve GR from 120 to 60 to control compute. Accuracy gains 0.4 points and training becomes 21% faster.\n\n**More transition layers.** The original DenseNet places transitions only between stages, also downsampling there. The authors add a stride-1 transition after every 3 blocks within a stage, solely to halve channels. The saved compute allows a larger GR, now varying by stage. This adds 1.5 percentage points, the largest single gain.\n\nThe final three steps are smaller. The stem becomes a 4×4, stride-4 patchify layer. Transitions replace average pooling with convolutions whose stride equals kernel size. Channel rescaling combines layer scale with effective SE. Each adds 0.1 to 0.2 points.\n\n[![Figure 1 from the paper: the upper left shows RDNet's overall sequence: Stem, Stage-1, Transition S/2, Stage-2, Transition S/2, Stage-3, Transition S/2, Stage-4, GlobalAvgPool, LayerNorm, FC. The upper right expands feature mixer f into DW Conv 7×7 (C→C), LayerNorm, Linear (C→4C), GeLU, Linear (4C→GR). The lower left expands Stage-N: purple feature blocks grow thicker by concatenation after each f, followed by a Transition S/1; the group repeats L_N times. The lower right shows Transition layer S/s as LayerNorm and an s×s stride-s convolution, halving channels.](/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png)](/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png)\n\n*Figure 1 from Kim et al., taken from [**PDF page 6**](https://arxiv.org/pdf/2403.19588v2#page=6). The original caption is omitted.*\n\nThis is the final RDNet. The lower half shows one stage: each feature mixer $f$ adds a purple slice. After three $f$ mixers, a stride-1 transition halves the thickness. This combination repeats $L_N$ times.\n\n### Following the Channel Counts\n\nThe figure alone makes channel growth hard to feel. timm provides an RDNet implementation and pretrained weights from `rdnet_tiny` to `rdnet_large`. Build an RDNet-T without downloading weights and print the channels in each block group:\n\n```python\nimport timm\nimport torch.nn as nn\n\nmodel = timm.create_model(\"rdnet_tiny\", pretrained=False)    # Architecture only; no weight download\nprint(\"stem output\", model.stem[0].out_channels, \"channels\")\nfor i, group in enumerate(model.dense_stages):\n    *transition, dense = group.children()\n    if transition:                                            # Transition between groups: LayerNorm + convolution\n        conv = transition[1]\n        kind = \"downsample\" if conv.stride[0] == 2 else \"compress channels only\"\n        print(f\"  Transition ({kind}) {conv.in_channels} → {conv.out_channels}\")\n    blocks = list(dense.children())\n    c_in = blocks[0].layers.layers[0].in_channels             # Input channels of the first block's 7×7 depthwise convolution\n    gr = blocks[0].growth_rate\n    print(f\"Group {i}: {len(blocks)} blocks, each appending {gr} channels  {c_in} → {c_in + len(blocks) * gr}\")\n```\n\n```text\nstem output 64 channels\nGroup 0: 3 blocks, each appending 64 channels  64 → 256\n  Transition (downsample) 256 → 128\nGroup 1: 3 blocks, each appending 104 channels  128 → 440\n  Transition (downsample) 440 → 216\nGroup 2: 3 blocks, each appending 128 channels  216 → 600\n  Transition (compress channels only) 600 → 296\nGroup 3: 3 blocks, each appending 128 channels  296 → 680\n  Transition (compress channels only) 680 → 336\nGroup 4: 3 blocks, each appending 128 channels  336 → 720\n  Transition (compress channels only) 720 → 360\nGroup 5: 3 blocks, each appending 128 channels  360 → 744\n  Transition (downsample) 744 → 368\nGroup 6: 3 blocks, each appending 224 channels  368 → 1040\n```\n\ntimm wraps every 3 blocks into a group. Groups 2 through 5 together form the paper's third stage, with 12 blocks. Channels grow within each group, then a transition halves them. Stage 3 therefore keeps moving between 216 and 744 channels.\n\nWhat if stage 3 had no internal transitions? Estimate it using the same numbers. Each block's costliest parts are the two 1×1 convolutions: expand to 4 times the input width, then compress to GR. Together, they take roughly $4c^2 + 4c\\cdot\\mathrm{GR}$ multiply-adds per spatial position:\n\n```python\ndef stage3(every, c=216, gr=128, blocks=12):\n    macs = 0\n    for b in range(blocks):\n        if b and b % every == 0:\n            c = int(c * 0.5 / 8) * 8                          # Transition halves the channel count\n        macs += c * 4 * c + 4 * c * gr                        # Per position: 1×1 expansion to 4×, then compression to GR\n        c += gr                                               # Append the output channels\n    return macs, c\n\nfor every, label in ((3, \"Compress every 3 blocks\"), (6, \"Compress every 6 blocks\"), (12, \"No compression within the stage\")):\n    macs, c = stage3(every)\n    print(f\"{label}: {c} final channels, {macs / 1e6:.1f}M multiply-adds per position for the two 1×1 convolutions\")\n```\n\n```text\nCompress every 3 blocks: 744 final channels, 12.2M multiply-adds per position for the two 1×1 convolutions\nCompress every 6 blocks: 1256 final channels, 29.0M multiply-adds per position for the two 1×1 convolutions\nNo compression within the stage: 1752 final channels, 55.7M multiply-adds per position for the two 1×1 convolutions\n```\n\nThis estimate in the note omits depthwise convolution and SE, counting only the largest terms. Without internal transitions, stage 3 ends at 1752 channels and takes 4.6 times the compute of compression every 3 blocks. The expansion layer's cost grows quadratically with channel count. Concatenation grows channels linearly, sending compute up quadratically. Frequent compression makes a larger GR affordable.\n\nThe paper's ablations support this interval too. At similar compute, compression every 2 blocks gives 82.7%, every 3 gives 82.8%, every 4 gives 82.6%, and every 6 falls to 82.1%.\n\n## Discussion\n\n### ImageNet\n\nFirst, compare representative models trained only on ImageNet-1K. Latency measures inference on 128 images at once; memory uses batch size 16:\n\n| Model | Parameters | FLOPs | Latency (ms) | Memory (GB) | top-1 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| Swin-T | 28M | 4.5G | 173 | 2.6 | 81.3% |\n| ConvNeXt-T | 29M | 4.5G | 150 | 2.7 | 82.1% |\n| InceptionNeXt-T | 28M | 4.2G | 132 | 3.3 | 82.3% |\n| RDNet-T | 24M | 5.0G | 175 | 4.1 | **82.8%** |\n| ConvNeXt-S | 50M | 8.7G | 266 | 4.0 | 83.1% |\n| RDNet-S | 50M | 8.7G | 289 | 5.4 | **83.7%** |\n| ConvNeXt-B | 89M | 15.4G | 417 | 5.4 | 83.8% |\n| RDNet-B | 87M | 15.4G | 472 | 6.9 | **84.4%** |\n\n*Table: selected rows from the paper's Table 4.*\n\nAt each size, RDNet gains 0.6 to 0.7 percentage points over ConvNeXt, at the cost of roughly 10% to 20% greater inference latency and 30% to 50% more memory. Keeping all preceding features still makes concatenation memory-hungry.\n\nThe authors also compare with newer models from 2022 to 2024: HorNet, VAN, BiFormer, NAT, SMT, and MogaNet. Here the tradeoff reverses. RDNet-T is similarly sized, with accuracy matching them or trailing by up to 1 percentage point, but batch-1 latency is just 7.4 ms, against 20 to 52 ms for the others. The table's description explicitly says their model trades a slight accuracy deficit for speed.\n\n### Downstream Tasks\n\nThe advantage is clearer in segmentation and detection. On ADE20K with UperNet, RDNet-T gets 47.6 single-scale mIoU against ConvNeXt-T's 46.0; at size B, it is 49.6 versus 49.1. On COCO with Mask R-CNN trained on a 3× schedule, RDNet-T reaches 47.5 box AP against ConvNeXt-T's 46.2, with 5M fewer parameters. The authors suggest that retaining features from different layers particularly helps dense prediction.\n\nThe authors also train CLIP with OpenCLIP, using RDNet-B and ConvNeXt-B as image encoders. ImageNet zero-shot top-1 is 54.1% versus 51.2%.\n\n### Stochastic Depth Helps Too\n\nThe original DenseNet did not use stochastic depth; the authors say their connection patterns already share some similarities. Testing it gives a substantial effect: a rate of 0 yields 81.6%, while 0.15 yields 82.8%, gaining 1.2 percentage points.\n\n\n> [!TIP]\n> \n> For how stochastic depth randomly lets a layer take the day off, see:\n> \n> - **[[16.03] Stochastic Depth: This Layer Is Taking the Day Off](/blog/papers/classic-cnns/stochastic-depth)**\n\n\n## Conclusion\n\nThe main results:\n\n1. **Concatenation itself**: over fifteen thousand random networks, concatenation gains 0.7 to 1.7 percentage points on average, using about 10% to 20% more memory.\n2. **DenseNet's proportions make it competitive**: becoming wider and shallower makes it fast; more transitions allow a larger GR. The latter step alone gains 1.5 points.\n3. **Results**: at similar size, accuracy beats ConvNeXt by 0.6 to 0.7 points, with larger gaps in segmentation and detection. Inference is somewhat slower and memory use 30% to 50% greater.\n\nSeen this way, DenseNet's neglect owed much to its deep, narrow proportions and old training recipe. Concatenation itself held up. Its appetite for memory, though, has stayed with it since 2016.\n","imageSizes":{"/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png":{"width":1230,"height":615}}}