{"notes":[{"id":"classic-cnns/rdnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RDNet","subtitle":"别加，接起来","author":"Z. Yuan","paperDate":"2024-03-28","publishedOn":"2026-01-14","world":"external/rdnet","paperTitle":"DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs","paperUrl":"https://arxiv.org/abs/2403.19588","bibliography":{"title":"DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs","authors":["Donghyun Kim","Byeongho Heo","Dongyoon Han"],"url":"https://arxiv.org/abs/2403.19588","pdfUrl":"https://arxiv.org/pdf/2403.19588v2","codeUrl":"https://github.com/naver-ai/rdnet","preprint":{"id":"2403.19588","submittedOn":"2024-03-28","url":"https://arxiv.org/abs/2403.19588"},"publication":{"venue":"ECCV","year":2024,"kind":"conference","url":"https://doi.org/10.1007/978-3-031-72646-0_23"},"citation":"@misc{classiccnnsrdnet2024,\n  title = {{DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs}},\n  author = {Donghyun Kim and Byeongho Heo and Dongyoon Han},\n  year = {2024},\n  eprint = {2403.19588},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2403.19588}\n}\n","citationPath":"/content/papers/citations/classic-cnns/rdnet.0ac4484543c0.bib"},"thumbnail":{"src":"/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png","width":1230,"height":615},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/rdnet.367cbc3244ea.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"加法和拼接差在哪","text":"加法和拼接差在哪","depth":3},{"id":"先抽一万五千个随机网络","text":"先抽一万五千个随机网络","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"从-densenet-201-一路改过来","text":"从 DenseNet-201 一路改过来","depth":3},{"id":"跟着通道数走一遍","text":"跟着通道数走一遍","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任务","text":"下游任务","depth":3},{"id":"stochastic-depth-也有用","text":"stochastic depth 也有用","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/densenet","relation":"referenced","topic":""},{"id":"classic-cnns/cspnet","relation":"referenced","topic":""},{"id":"classic-cnns/regnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/densenet","category":"classic-cnns","legacyPath":"classic-cnns/1608-densenet","legacySlug":"1608-densenet","legacyPaths":[],"title":"DenseNet","subtitle":"全部连起来","author":"Z. Yuan","paperDate":"2016-08-25","publishedOn":"2024-07-02","paperTitle":"Densely Connected Convolutional Networks","paperUrl":"https://arxiv.org/abs/1608.06993","bibliography":{"title":"Densely Connected Convolutional Networks","authors":["Gao Huang","Zhuang Liu","Laurens van der Maaten","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1608.06993","pdfUrl":"https://arxiv.org/pdf/1608.06993","codeUrl":null,"preprint":{"id":"1608.06993","submittedOn":"2016-08-25","url":"https://arxiv.org/abs/1608.06993"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1608.06993"},"citation":"@misc{classiccnnsdensenet2016,\n  title = {{Densely Connected Convolutional Networks}},\n  author = {Gao Huang and Zhuang Liu and Laurens van der Maaten and Kilian Q. Weinberger},\n  year = {2016},\n  eprint = {1608.06993},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1608.06993}\n}\n","citationPath":"/content/papers/citations/classic-cnns/densenet.6f28153b67f7.bib"},"thumbnail":{"src":"/content/papers/assets/4f073a0fd1fc5b7415ca19c48934d180f9043fddc08301819a50ceb1b0ebfc69.jpg","width":1184,"height":828},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/densenet.c4f2eed4ee7c.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"网络架构","text":"网络架构","depth":3},{"id":"平均池化","text":"平均池化","depth":3},{"id":"growth-rate","text":"Growth Rate","depth":3},{"id":"bottleneck-layers","text":"Bottleneck Layers","depth":3},{"id":"compression","text":"Compression","depth":3},{"id":"实作细节","text":"实作细节","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验结果","text":"实验结果","depth":3},{"id":"紧凑模型","text":"紧凑模型","depth":3},{"id":"特征重用","text":"特征重用","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"background","topic":"如何训练更深的卷积网络"}]},{"id":"classic-cnns/cspnet","category":"classic-cnns","legacyPath":"classic-cnns/1911-cspnet","legacySlug":"1911-cspnet","legacyPaths":[],"title":"CSPNet","subtitle":"半半半插件","author":"Z. Yuan","paperDate":"2019-11-27","publishedOn":"2024-09-29","paperTitle":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","paperUrl":"https://arxiv.org/abs/1911.11929","bibliography":{"title":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","authors":["Chien-Yao Wang","Hong-Yuan Mark Liao","I-Hau Yeh","Yueh-Hua Wu","Ping-Yang Chen","Jun-Wei Hsieh"],"url":"https://arxiv.org/abs/1911.11929","pdfUrl":"https://arxiv.org/pdf/1911.11929","codeUrl":null,"preprint":{"id":"1911.11929","submittedOn":"2019-11-27","url":"https://arxiv.org/abs/1911.11929"},"publication":{"venue":"CVPR Workshops","year":2020,"kind":"workshop","url":"https://doi.org/10.1109/cvprw50498.2020.00203"},"citation":"@misc{classiccnnscspnet2019,\n  title = {{CSPNet: A New Backbone that can Enhance Learning Capability of CNN}},\n  author = {Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},\n  year = {2019},\n  eprint = {1911.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1911.11929}\n}\n","citationPath":"/content/papers/citations/classic-cnns/cspnet.2914b9492a0c.bib"},"thumbnail":{"src":"/content/papers/assets/2602af8b5f92cba820e545c84abd94943cab1fe2631cf83834972b54b374ed20.jpg","width":2809,"height":1911},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/cspnet.49f7ed2a19fe.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型架构","text":"模型架构","depth":3},{"id":"transition-layer","text":"Transition Layer","depth":3},{"id":"exact-fusion-model","text":"Exact Fusion Model","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"csp-架构的有效性","text":"CSP 架构的有效性","depth":3},{"id":"efm-架构的有效性","text":"EFM 架构的有效性","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"ms-coco-上的性能","text":"MS COCO 上的性能","depth":3},{"id":"推理速度","text":"推理速度","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分组卷积与特征复用"},{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分组卷积与特征复用"}]},{"id":"classic-cnns/regnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RegNet","subtitle":"好架构都很像吗？","author":"Z. Yuan","paperDate":"2020-03-30","publishedOn":"2024-12-16","world":"external/regnet","paperTitle":"Designing Network Design Spaces","paperUrl":"https://arxiv.org/abs/2003.13678","bibliography":{"title":"Designing Network Design Spaces","authors":["Ilija Radosavovic","Raj Prateek Kosaraju","Ross Girshick","Kaiming He","Piotr Dollár"],"url":"https://arxiv.org/abs/2003.13678","pdfUrl":"https://arxiv.org/pdf/2003.13678v1","codeUrl":null,"preprint":{"id":"2003.13678","submittedOn":"2020-03-30","url":"https://arxiv.org/abs/2003.13678"},"publication":{"venue":"CVPR","year":2020,"kind":"conference","url":"https://openaccess.thecvf.com/content_CVPR_2020/html/Radosavovic_Designing_Network_Design_Spaces_CVPR_2020_paper.html"},"citation":"@misc{classiccnnsregnet2020,\n  title = {{Designing Network Design Spaces}},\n  author = {Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Dollár},\n  year = {2020},\n  eprint = {2003.13678},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2003.13678}\n}\n","citationPath":"/content/papers/citations/classic-cnns/regnet.51d4a967c43d.bib"},"thumbnail":{"src":"/content/papers/assets/477f84998189b88eda3ae1c093bfad949ef34cf53e00368019c348b971c3a63d.png","width":1013,"height":550},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/regnet.f79de4cb64b2.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"手工设计和-nas-各缺一块","text":"手工设计和 NAS 各缺一块","depth":3},{"id":"怎么比较两个设计空间","text":"怎么比较两个设计空间","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"起点anynet","text":"起点：AnyNet","depth":3},{"id":"一步步收窄","text":"一步步收窄","depth":3},{"id":"宽度是一条斜线","text":"宽度是一条斜线","depth":3},{"id":"regnet","text":"RegNet","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"和常见做法不一样的地方","text":"和常见做法不一样的地方","depth":3},{"id":"和-resnetresnext-比","text":"和 ResNet、ResNeXt 比","depth":3},{"id":"移动设备的计算量","text":"移动设备的计算量","depth":3},{"id":"和-efficientnet-比","text":"和 EfficientNet 比","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/nasnet","relation":"referenced","topic":""},{"id":"classic-cnns/efficientnet","relation":"referenced","topic":""}]}],"text":"捷径连接有两种做法。ResNet 把输入加到输出上，$x + f(x)$；DenseNet 把输入和输出接在一起，$[x, f(x)]$。\n\n2016 年的 DenseNet 用更少的参数追上了 ResNet，拿下 CVPR 2017 的最佳论文。可是后来的主流网络，ResNeXt、EfficientNet、ConvNeXt，再到 ViT 和 Swin，用的几乎都是加法。DenseNet 慢慢淡出了主流架构的讨论。\n\n这篇来自 NAVER Cloud AI 和 NAVER AI Lab，作者是 Donghyun Kim、Byeongho Heo 和 Dongyoon Han。他们认为 DenseNet 被冷落，问题不在拼接本身，而在过时的训练方法和老旧的设计细节。所以他们保留拼接，其他全部翻新，看看 DenseNet 能走多远。\n\n\n> [!TIP]\n> \n> DenseNet 的原始设计，以及后来借用它部分概念的 CSPNet，可以先看这两篇：\n> \n> - **[[16.08] DenseNet：全部连起来](/blog/papers/classic-cnns/densenet)**\n> - **[[19.11] CSPNet：半半半插件](/blog/papers/classic-cnns/cspnet)**\n\n\n## 定义问题\n\n### 加法和拼接差在哪\n\n加法捷径的输出宽度固定。不管叠几层，特征始终是 $d$ 个通道，每一层只能修改这 $d$ 个通道的内容。\n\n拼接捷径每过一层，通道数就多 $g$ 个，$g$ 叫做增长率（growth rate，以下简称 GR）。前面所有层的特征都原封不动地留着，后面的层可以直接拿来用。\n\n作者用矩阵的秩来说明拼接的好处。一层的输出 $f(XW)$，$X$ 是 $N\\times d_{in}$ 的输入，$W$ 是 $d_{in}\\times d_{out}$ 的权重，有了非线性之后，输出的秩通常会接近 $d_{out}$。输出维度越高，表达能力越强。拼接等于把权重写成 $W=[W_P, I]$，原本的 $d_{in}$ 个通道保留，再多 $W_P$ 产生的 $g$ 个，秩就一路往上加。\n\n这也是 DenseNet 的老毛病所在。通道一直变多，内存一直涨，所以原版 DenseNet 只敢用很小的 GR（例如 32），然后把网络叠得很深，DenseNet-201 有四个密集区块，各有 6、12、48、32 层。又深又窄的网络，在 GPU 上跑得很慢。\n\n作者的另一个论点是，用过渡层把通道压小，秩不一定会掉很多，只要压完的维度不要太小。所以频繁地压，可以控制内存，又不太伤表达能力。\n\n### 先抽一万五千个随机网络\n\n这些还只是推测，作者先用实验验证。他们在 Tiny-ImageNet 上随机产生超过 1.5 万个网络，分成加法捷径和拼接捷径两组，深度、宽度、激活函数、归一化、卷积核大小也都随机。只有算力落在预算内的网络才拿来训练，两种捷径的计算成本因此差不多。\n\n随机空间分成五种，A、B、C 是三种大小，D 在 C 的基础上加数据增强，E 再把优化器换成 AdamW：\n\n| 空间 | 加法 top-1 | 拼接 top-1 | 加法内存（GB） | 拼接内存（GB） |\n| --- | ---: | ---: | ---: | ---: |\n| A（约 2.2M 参数） | 45.8 ± 2.0% | **47.5 ± 2.1%** | 0.65 | 0.75 |\n| B（约 4.5M） | 50.1 ± 2.1% | **51.2 ± 2.1%** | 0.78 | 0.90 |\n| C（约 9.5M） | 53.2 ± 2.2% | **54.3 ± 2.0%** | 1.01 | 1.24 |\n| D（C 加数据增强） | 57.4 ± 1.5% | **58.1 ± 1.4%** | 1.02 | 1.24 |\n| E（D 换成 AdamW） | 58.2 ± 1.5% | **58.9 ± 1.6%** | 1.02 | 1.25 |\n\n*表：摘自原论文 Table 7，每个空间内所有网络的平均与标准差。*\n\n五个空间里，拼接的平均都比加法高，差距在 0.7 到 1.7 个百分点之间。不过标准差有 2 个百分点左右，两边的分布其实重叠很多；训练配方变强之后，差距也从 1.1 缩到 0.7。另一边，拼接的内存多了 15% 到 23%，DenseNet 的老毛病还在。\n\n这组实验用的是 64×64 的小图和不到 1,000 万参数的小网络，给的是一个方向，还不足以说明大网络也会这样。所以接下来要在 ImageNet 上真的做一次。\n\n\n> [!TIP]\n> \n> 「随机抽一大堆网络，比较准确率分布」这个方法来自 RegNet 的作者，可以看这篇：\n> \n> - **[[20.03] RegNet：好架构都很像吗？](/blog/papers/classic-cnns/regnet)**\n\n\n## 解决问题\n\n### 从 DenseNet-201 一路改过来\n\n作者的做法和 ConvNeXt 很像，从一个旧模型出发，一次改一个地方，记下每一步的准确率和速度。起点是 DenseNet-201，先换上现代的训练配方，AdamW、cosine 学习率、300 个 epoch、各种数据增强，ImageNet 是 79.7%。\n\n| 步骤 | top-1 | 参数量 | FLOPs | 推理延迟（批量 1） | 训练延迟 | 训练内存 |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| DenseNet-201（新配方） | 79.7% | 20.0M | 4.3G | 38.4 ms | 131 ms | 3.9 GB |\n| 加宽、变浅 | 79.5% | 21.8M | 11.1G | 8.5 ms | 85 ms | 3.2 GB |\n| 换成现代的区块 | 80.4% | 12.9M | 4.8G | 10.4 ms | 112 ms | 3.4 GB |\n| 扩张倍数改跟着输入走 | 80.8% | 19.9M | 4.7G | 11.8 ms | 88 ms | 3.1 GB |\n| 更多过渡层、各阶段不同 GR | 82.3% | 21.2M | 5.0G | 11.0 ms | 90 ms | 3.4 GB |\n| patchify stem | 82.4% | 21.2M | 4.9G | 11.0 ms | 88 ms | 3.2 GB |\n| 重新设计过渡层 | 82.6% | 22.4M | 4.9G | 13.6 ms | 97 ms | 3.1 GB |\n| 通道重新缩放 | **82.8%** | 23.9M | 5.0G | 14.0 ms | 99 ms | 3.1 GB |\n\n*表：摘自原论文 Table 1 的部分列。*\n\n一步一步看。\n\n**加宽、变浅。** GR 从 32 拉到 120，四个阶段的区块数从 $(6, 12, 48, 32)$ 砍到 $(3, 3, 12, 3)$。准确率几乎没变，训练延迟少了 35%，内存少了 18%。FLOPs 涨到 11.1G，后面再处理。\n\n**换成现代的区块。** 每个区块改成 ConvNeXt 的样子，7×7 depthwise 卷积、LayerNorm、1×1 卷积扩张再压回来，归一化和激活函数也减少。唯一的差别是输出通道是 GR，比输入的通道数少，每个区块都在把前面接起来的特征压缩成 GR 个新通道。这一步多了 0.9 个百分点。\n\n**扩张倍数跟着输入走。** 原版 DenseNet 的瓶颈层宽度是 GR 的 4 倍。作者改成输入通道数的 4 倍，让非线性之前的特征更丰富，再把 GR 从 120 减半到 60，控制计算量。准确率多 0.4，训练还快了 21%。\n\n**更多过渡层。** 原版 DenseNet 只在阶段之间放过渡层，顺便降采样。作者在阶段里面每 3 个区块就放一个步长 1 的过渡层，只负责把通道压成一半。省下来的计算量拿去把 GR 调高，各阶段的 GR 也不再一样。这一步多了 1.5 个百分点，是整个流程里最大的一步。\n\n最后三步比较小。stem 改成 4×4、步长 4 的 patchify；过渡层拿掉平均池化，直接用步长等于卷积核大小的卷积降采样；再加上通道重新缩放（layer scale 加上 effective SE），各多 0.1 到 0.2。\n\n[![原论文 Fig. 1：上半部左边是 RDNet 的整体流程，依序是 Stem、Stage-1、Transition S/2、Stage-2、Transition S/2、Stage-3、Transition S/2、Stage-4、GlobalAvgPool、LayerNorm、FC；右边是 Feature mixer f，依序是 DW Conv 7×7（C→C）、LayerNorm、Linear（C→4C）、GeLU、Linear（4C→GR）。下半部左边展开 Stage-N，紫色的特征方块一路拼接变厚，每经过一个 f 就多一片，最后接一个 Transition S/1，整组重复 L_N 次；右边是 Transition layer S/s，由 LayerNorm 和步长 s 的 s×s 卷积组成，通道数减半。](/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png)](/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png)\n\n*原论文 Fig. 1（Kim 等人），取自 [**PDF 第 6 页**](https://arxiv.org/pdf/2403.19588v2#page=6)。原图注未收录。*\n\n最后的 RDNet 长这样。下半部是一个阶段的内部，紫色方块每经过一个特征混合器 $f$ 就多一片，三个 $f$ 之后接一个步长 1 的过渡层，把厚度压回一半，这样的组合重复 $L_N$ 次。\n\n### 跟着通道数走一遍\n\n光看图不容易感觉到通道数怎么变化。timm 里有 RDNet 的实现和 `rdnet_tiny` 到 `rdnet_large` 的预训练权重，我们直接建一个 RDNet-T（不下载权重），打印每组区块的通道数：\n\n```python\nimport timm\nimport torch.nn as nn\n\nmodel = timm.create_model(\"rdnet_tiny\", pretrained=False)    # 只要架构，不下载权重\nprint(\"stem 输出\", model.stem[0].out_channels, \"个通道\")\nfor i, group in enumerate(model.dense_stages):\n    *transition, dense = group.children()\n    if transition:                                            # 组与组之间的过渡层：LayerNorm + 卷积\n        conv = transition[1]\n        kind = \"降采样\" if conv.stride[0] == 2 else \"只压通道\"\n        print(f\"  过渡层（{kind}） {conv.in_channels} → {conv.out_channels}\")\n    blocks = list(dense.children())\n    c_in = blocks[0].layers.layers[0].in_channels             # 第一个区块的 7×7 depthwise 吃进多少通道\n    gr = blocks[0].growth_rate\n    print(f\"第 {i} 组 {len(blocks)} 个区块，每块接上 {gr} 个通道  {c_in} → {c_in + len(blocks) * gr}\")\n```\n\n```text\nstem 输出 64 个通道\n第 0 组 3 个区块，每块接上 64 个通道  64 → 256\n  过渡层（降采样） 256 → 128\n第 1 组 3 个区块，每块接上 104 个通道  128 → 440\n  过渡层（降采样） 440 → 216\n第 2 组 3 个区块，每块接上 128 个通道  216 → 600\n  过渡层（只压通道） 600 → 296\n第 3 组 3 个区块，每块接上 128 个通道  296 → 680\n  过渡层（只压通道） 680 → 336\n第 4 组 3 个区块，每块接上 128 个通道  336 → 720\n  过渡层（只压通道） 720 → 360\n第 5 组 3 个区块，每块接上 128 个通道  360 → 744\n  过渡层（降采样） 744 → 368\n第 6 组 3 个区块，每块接上 224 个通道  368 → 1040\n```\n\ntimm 把每 3 个区块包成一组，第 2 到第 5 组合起来就是论文说的第三阶段，12 个区块。每组结束时通道变多，过渡层再压回一半，所以第三阶段的宽度一直在 216 到 744 之间来回。\n\n如果第三阶段中间不放过渡层呢？我们用同样的数字估一下。每个区块最贵的是两个 1×1 卷积，先扩张到输入的 4 倍，再压回 GR，这两个卷积每个位置的乘加数大约是 $4c^2 + 4c\\cdot\\mathrm{GR}$：\n\n```python\ndef stage3(every, c=216, gr=128, blocks=12):\n    macs = 0\n    for b in range(blocks):\n        if b and b % every == 0:\n            c = int(c * 0.5 / 8) * 8                          # 过渡层把通道压成一半\n        macs += c * 4 * c + 4 * c * gr                        # 每个位置：1×1 扩张到 4 倍，再压回 GR\n        c += gr                                               # 输出接到后面\n    return macs, c\n\nfor every, label in ((3, \"每 3 块压一次\"), (6, \"每 6 块压一次\"), (12, \"整段都不压\")):\n    macs, c = stage3(every)\n    print(f\"{label}：最后 {c} 个通道，两个 1×1 卷积每个位置 {macs / 1e6:.1f}M 次乘加\")\n```\n\n```text\n每 3 块压一次：最后 744 个通道，两个 1×1 卷积每个位置 12.2M 次乘加\n每 6 块压一次：最后 1256 个通道，两个 1×1 卷积每个位置 29.0M 次乘加\n整段都不压：最后 1752 个通道，两个 1×1 卷积每个位置 55.7M 次乘加\n```\n\n这里省略了 depthwise 卷积和 SE，只算最大头的部分。不放过渡层，第三阶段最后会长到 1752 个通道，计算量是每 3 块压一次的 4.6 倍。因为扩张层的成本和通道数的平方成正比，拼接让通道线性增长，计算量就跟着平方增长。频繁地压，才有本钱把 GR 调高。\n\n论文的消融也支持这个间隔。在差不多的计算量下，每 2 块压一次是 82.7%，每 3 块 82.8%，每 4 块 82.6%，每 6 块掉到 82.1%。\n\n## 讨论\n\n### ImageNet\n\n先和几个代表性的模型比，全部只用 ImageNet-1K 训练，延迟是一次推理 128 张图，内存用批量 16 量：\n\n| 模型 | 参数量 | FLOPs | 延迟（ms） | 内存（GB） | top-1 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| Swin-T | 28M | 4.5G | 173 | 2.6 | 81.3% |\n| ConvNeXt-T | 29M | 4.5G | 150 | 2.7 | 82.1% |\n| InceptionNeXt-T | 28M | 4.2G | 132 | 3.3 | 82.3% |\n| RDNet-T | 24M | 5.0G | 175 | 4.1 | **82.8%** |\n| ConvNeXt-S | 50M | 8.7G | 266 | 4.0 | 83.1% |\n| RDNet-S | 50M | 8.7G | 289 | 5.4 | **83.7%** |\n| ConvNeXt-B | 89M | 15.4G | 417 | 5.4 | 83.8% |\n| RDNet-B | 87M | 15.4G | 472 | 6.9 | **84.4%** |\n\n*表：摘自原论文 Table 4 的部分行。*\n\nRDNet 每个尺寸都比 ConvNeXt 多 0.6 到 0.7 个百分点，代价是推理延迟多了大约一到两成，内存也多了大约三到五成。拼接要保留前面所有的特征，内存还是比较吃。\n\n作者另外拿 2022 到 2024 年的新模型来比，HorNet、VAN、BiFormer、NAT、SMT、MogaNet。这时情况反过来，RDNet-T 和它们差不多大，准确率持平或低了最多 1 个百分点，批量 1 的延迟却只有 7.4 ms，其他模型在 20 到 52 ms 之间。作者在表的说明里也直接写了，他们的模型准确率稍微落后，靠速度补回来。\n\n### 下游任务\n\n分割和检测的优势比较明显。ADE20K 上接 UperNet，RDNet-T 的单尺度 mIoU 是 47.6，ConvNeXt-T 是 46.0；B 是 49.6 对 49.1。COCO 上用 Mask R-CNN 训练 3 倍训练计划，RDNet-T 的 box AP 是 47.5，ConvNeXt-T 是 46.2，参数还少了 5M。作者认为，拼接保留了不同层的特征，这对密集预测特别有用。\n\n作者也用 OpenCLIP 训练了 CLIP，图像编码器分别用 RDNet-B 和 ConvNeXt-B，ImageNet 零样本分类的 top-1 是 54.1% 对 51.2%。\n\n### stochastic depth 也有用\n\n原版 DenseNet 没有用 stochastic depth，作者说是因为两者的连接模式本来就有相似之处。实际试了之后，影响很大，比例 0 是 81.6%，0.15 是 82.8%，多了 1.2 个百分点。\n\n\n> [!TIP]\n> \n> stochastic depth 是怎么随机让一层「请假」的，可以看这篇：\n> \n> - **[[16.03] Stochastic Depth：今天这层请假](/blog/papers/classic-cnns/stochastic-depth)**\n\n\n## 结论\n\n最后整理一下：\n\n1. **拼接本身**：一万五千个随机网络里，拼接平均比加法高 0.7 到 1.7 个百分点，内存也多了一到两成。\n2. **让 DenseNet 能打的是身材**：加宽变浅让它变快，更多过渡层让它能把 GR 调高，后者一步就多了 1.5 个百分点。\n3. **成绩**：同样大小比 ConvNeXt 准 0.6 到 0.7 个百分点，分割和检测的差距更大；推理慢一些，内存多三到五成。\n\n这样看来，DenseNet 当年被冷落，问题大多出在又深又窄的身材和旧的训练配方，拼接本身倒是没做错什么。只是它要的内存，从 2016 年到现在都没少过。\n","imageSizes":{"/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png":{"width":1230,"height":615}}}