{"notes":[{"id":"classic-cnns/rdnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RDNet","subtitle":"別加，接起來","author":"Z. Yuan","paperDate":"2024-03-28","publishedOn":"2026-01-14","world":"external/rdnet","paperTitle":"DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs","paperUrl":"https://arxiv.org/abs/2403.19588","bibliography":{"title":"DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs","authors":["Donghyun Kim","Byeongho Heo","Dongyoon Han"],"url":"https://arxiv.org/abs/2403.19588","pdfUrl":"https://arxiv.org/pdf/2403.19588v2","codeUrl":"https://github.com/naver-ai/rdnet","preprint":{"id":"2403.19588","submittedOn":"2024-03-28","url":"https://arxiv.org/abs/2403.19588"},"publication":{"venue":"ECCV","year":2024,"kind":"conference","url":"https://doi.org/10.1007/978-3-031-72646-0_23"},"citation":"@misc{classiccnnsrdnet2024,\n  title = {{DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs}},\n  author = {Donghyun Kim and Byeongho Heo and Dongyoon Han},\n  year = {2024},\n  eprint = {2403.19588},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2403.19588}\n}\n","citationPath":"/content/papers/citations/classic-cnns/rdnet.0ac4484543c0.bib"},"thumbnail":{"src":"/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png","width":1230,"height":615},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/rdnet.35b79ae3df83.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"加法和串接差在哪","text":"加法和串接差在哪","depth":3},{"id":"先抽一萬五千個隨機網路","text":"先抽一萬五千個隨機網路","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"從-densenet-201-一路改過來","text":"從 DenseNet-201 一路改過來","depth":3},{"id":"跟著通道數走一遍","text":"跟著通道數走一遍","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet","text":"ImageNet","depth":3},{"id":"下游任務","text":"下游任務","depth":3},{"id":"stochastic-depth-也有用","text":"stochastic depth 也有用","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/densenet","relation":"referenced","topic":""},{"id":"classic-cnns/cspnet","relation":"referenced","topic":""},{"id":"classic-cnns/regnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/densenet","category":"classic-cnns","legacyPath":"classic-cnns/1608-densenet","legacySlug":"1608-densenet","legacyPaths":[],"title":"DenseNet","subtitle":"全部連起來","author":"Z. Yuan","paperDate":"2016-08-25","publishedOn":"2024-07-02","paperTitle":"Densely Connected Convolutional Networks","paperUrl":"https://arxiv.org/abs/1608.06993","bibliography":{"title":"Densely Connected Convolutional Networks","authors":["Gao Huang","Zhuang Liu","Laurens van der Maaten","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1608.06993","pdfUrl":"https://arxiv.org/pdf/1608.06993","codeUrl":null,"preprint":{"id":"1608.06993","submittedOn":"2016-08-25","url":"https://arxiv.org/abs/1608.06993"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1608.06993"},"citation":"@misc{classiccnnsdensenet2016,\n  title = {{Densely Connected Convolutional Networks}},\n  author = {Gao Huang and Zhuang Liu and Laurens van der Maaten and Kilian Q. Weinberger},\n  year = {2016},\n  eprint = {1608.06993},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1608.06993}\n}\n","citationPath":"/content/papers/citations/classic-cnns/densenet.6f28153b67f7.bib"},"thumbnail":{"src":"/content/papers/assets/4f073a0fd1fc5b7415ca19c48934d180f9043fddc08301819a50ceb1b0ebfc69.jpg","width":1184,"height":828},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/densenet.1ac5b17fc8c5.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"網路架構","text":"網路架構","depth":3},{"id":"平均池化","text":"平均池化","depth":3},{"id":"growth-rate","text":"Growth Rate","depth":3},{"id":"bottleneck-layers","text":"Bottleneck Layers","depth":3},{"id":"compression","text":"Compression","depth":3},{"id":"實作細節","text":"實作細節","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗結果","text":"實驗結果","depth":3},{"id":"緊湊模型","text":"緊湊模型","depth":3},{"id":"特徵重用","text":"特徵重用","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"background","topic":"如何訓練更深的卷積網路"}]},{"id":"classic-cnns/cspnet","category":"classic-cnns","legacyPath":"classic-cnns/1911-cspnet","legacySlug":"1911-cspnet","legacyPaths":[],"title":"CSPNet","subtitle":"半半半插件","author":"Z. Yuan","paperDate":"2019-11-27","publishedOn":"2024-09-29","paperTitle":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","paperUrl":"https://arxiv.org/abs/1911.11929","bibliography":{"title":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","authors":["Chien-Yao Wang","Hong-Yuan Mark Liao","I-Hau Yeh","Yueh-Hua Wu","Ping-Yang Chen","Jun-Wei Hsieh"],"url":"https://arxiv.org/abs/1911.11929","pdfUrl":"https://arxiv.org/pdf/1911.11929","codeUrl":null,"preprint":{"id":"1911.11929","submittedOn":"2019-11-27","url":"https://arxiv.org/abs/1911.11929"},"publication":{"venue":"CVPR Workshops","year":2020,"kind":"workshop","url":"https://doi.org/10.1109/cvprw50498.2020.00203"},"citation":"@misc{classiccnnscspnet2019,\n  title = {{CSPNet: A New Backbone that can Enhance Learning Capability of CNN}},\n  author = {Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},\n  year = {2019},\n  eprint = {1911.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1911.11929}\n}\n","citationPath":"/content/papers/citations/classic-cnns/cspnet.2914b9492a0c.bib"},"thumbnail":{"src":"/content/papers/assets/2602af8b5f92cba820e545c84abd94943cab1fe2631cf83834972b54b374ed20.jpg","width":2809,"height":1911},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/cspnet.df77c5a48e8b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"transition-layer","text":"Transition Layer","depth":3},{"id":"exact-fusion-model","text":"Exact Fusion Model","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"csp-架構的有效性","text":"CSP 架構的有效性","depth":3},{"id":"efm-架構的有效性","text":"EFM 架構的有效性","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"ms-coco-上的性能","text":"MS COCO 上的性能","depth":3},{"id":"推論速度","text":"推論速度","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分組卷積與特徵重用"},{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分組卷積與特徵重用"}]},{"id":"classic-cnns/regnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RegNet","subtitle":"好架構都很像嗎？","author":"Z. Yuan","paperDate":"2020-03-30","publishedOn":"2024-12-16","world":"external/regnet","paperTitle":"Designing Network Design Spaces","paperUrl":"https://arxiv.org/abs/2003.13678","bibliography":{"title":"Designing Network Design Spaces","authors":["Ilija Radosavovic","Raj Prateek Kosaraju","Ross Girshick","Kaiming He","Piotr Dollár"],"url":"https://arxiv.org/abs/2003.13678","pdfUrl":"https://arxiv.org/pdf/2003.13678v1","codeUrl":null,"preprint":{"id":"2003.13678","submittedOn":"2020-03-30","url":"https://arxiv.org/abs/2003.13678"},"publication":{"venue":"CVPR","year":2020,"kind":"conference","url":"https://openaccess.thecvf.com/content_CVPR_2020/html/Radosavovic_Designing_Network_Design_Spaces_CVPR_2020_paper.html"},"citation":"@misc{classiccnnsregnet2020,\n  title = {{Designing Network Design Spaces}},\n  author = {Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Dollár},\n  year = {2020},\n  eprint = {2003.13678},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2003.13678}\n}\n","citationPath":"/content/papers/citations/classic-cnns/regnet.51d4a967c43d.bib"},"thumbnail":{"src":"/content/papers/assets/477f84998189b88eda3ae1c093bfad949ef34cf53e00368019c348b971c3a63d.png","width":1013,"height":550},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/regnet.fa68d07aa71a.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"手工設計和-nas-各缺一塊","text":"手工設計和 NAS 各缺一塊","depth":3},{"id":"怎麼比較兩個設計空間","text":"怎麼比較兩個設計空間","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"起點anynet","text":"起點：AnyNet","depth":3},{"id":"一步步收窄","text":"一步步收窄","depth":3},{"id":"寬度是一條斜線","text":"寬度是一條斜線","depth":3},{"id":"regnet","text":"RegNet","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"和常見做法不一樣的地方","text":"和常見做法不一樣的地方","depth":3},{"id":"和-resnetresnext-比","text":"和 ResNet、ResNeXt 比","depth":3},{"id":"行動裝置的運算量","text":"行動裝置的運算量","depth":3},{"id":"和-efficientnet-比","text":"和 EfficientNet 比","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/nasnet","relation":"referenced","topic":""},{"id":"classic-cnns/efficientnet","relation":"referenced","topic":""}]}],"text":"捷徑連接有兩種做法。ResNet 把輸入加到輸出上，$x + f(x)$；DenseNet 把輸入和輸出接在一起，$[x, f(x)]$。\n\n2016 年的 DenseNet 用更少的參數追上了 ResNet，拿下 CVPR 2017 的最佳論文。可是後來的主流網路，ResNeXt、EfficientNet、ConvNeXt，再到 ViT 和 Swin，用的幾乎都是加法。DenseNet 慢慢淡出了主流架構的討論。\n\n這篇來自 NAVER Cloud AI 和 NAVER AI Lab，作者是 Donghyun Kim、Byeongho Heo 和 Dongyoon Han。他們認為 DenseNet 被冷落，問題不在串接本身，而在過時的訓練方法和老舊的設計細節。所以他們保留串接，其他全部翻新，看看 DenseNet 能走多遠。\n\n\n> [!TIP]\n> \n> DenseNet 的原始設計，以及後來借用它部分概念的 CSPNet，可以先看這兩篇：\n> \n> - **[[16.08] DenseNet：全部連起來](/blog/papers/classic-cnns/densenet)**\n> - **[[19.11] CSPNet：半半半插件](/blog/papers/classic-cnns/cspnet)**\n\n\n## 定義問題\n\n### 加法和串接差在哪\n\n加法捷徑的輸出寬度固定。不管疊幾層，特徵始終是 $d$ 個通道，每一層只能修改這 $d$ 個通道的內容。\n\n串接捷徑每過一層，通道數就多 $g$ 個，$g$ 叫做成長率（growth rate，以下簡稱 GR）。前面所有層的特徵都原封不動地留著，後面的層可以直接拿來用。\n\n作者用矩陣的秩來說明串接的好處。一層的輸出 $f(XW)$，$X$ 是 $N\\times d_{in}$ 的輸入，$W$ 是 $d_{in}\\times d_{out}$ 的權重，有了非線性之後，輸出的秩通常會接近 $d_{out}$。輸出維度越高，表達能力越強。串接等於把權重寫成 $W=[W_P, I]$，原本的 $d_{in}$ 個通道保留，再多 $W_P$ 產生的 $g$ 個，秩就一路往上加。\n\n這也是 DenseNet 的老毛病所在。通道一直變多，記憶體一直漲，所以原版 DenseNet 只敢用很小的 GR（例如 32），然後把網路疊得很深，DenseNet-201 有四個密集區塊，各有 6、12、48、32 層。又深又窄的網路，在 GPU 上跑得很慢。\n\n作者的另一個論點是，用過渡層把通道壓小，秩不一定會掉很多，只要壓完的維度不要太小。所以頻繁地壓，可以控制記憶體，又不太傷表達能力。\n\n### 先抽一萬五千個隨機網路\n\n這些還只是推測，作者先用實驗驗證。他們在 Tiny-ImageNet 上隨機產生超過 1.5 萬個網路，分成加法捷徑和串接捷徑兩組，深度、寬度、啟動函數、正規化、卷積核大小也都隨機。只有算力落在預算內的網路才拿來訓練，兩種捷徑的計算成本因此差不多。\n\n隨機空間分成五種，A、B、C 是三種大小，D 在 C 的基礎上加資料增強，E 再把優化器換成 AdamW：\n\n| 空間 | 加法 top-1 | 串接 top-1 | 加法記憶體（GB） | 串接記憶體（GB） |\n| --- | ---: | ---: | ---: | ---: |\n| A（約 2.2M 參數） | 45.8 ± 2.0% | **47.5 ± 2.1%** | 0.65 | 0.75 |\n| B（約 4.5M） | 50.1 ± 2.1% | **51.2 ± 2.1%** | 0.78 | 0.90 |\n| C（約 9.5M） | 53.2 ± 2.2% | **54.3 ± 2.0%** | 1.01 | 1.24 |\n| D（C 加資料增強） | 57.4 ± 1.5% | **58.1 ± 1.4%** | 1.02 | 1.24 |\n| E（D 換成 AdamW） | 58.2 ± 1.5% | **58.9 ± 1.6%** | 1.02 | 1.25 |\n\n*表：摘自原論文 Table 7，每個空間內所有網路的平均與標準差。*\n\n五個空間裡，串接的平均都比加法高，差距在 0.7 到 1.7 個百分點之間。不過標準差有 2 個百分點左右，兩邊的分布其實重疊很多；訓練配方變強之後，差距也從 1.1 縮到 0.7。另一邊，串接的記憶體多了 15% 到 23%，DenseNet 的老毛病還在。\n\n這組實驗用的是 64×64 的小圖和不到 1,000 萬參數的小網路，給的是一個方向，還不足以說明大網路也會這樣。所以接下來要在 ImageNet 上真的做一次。\n\n\n> [!TIP]\n> \n> 「隨機抽一大堆網路，比較準確率分布」這個方法來自 RegNet 的作者，可以看這篇：\n> \n> - **[[20.03] RegNet：好架構都很像嗎？](/blog/papers/classic-cnns/regnet)**\n\n\n## 解決問題\n\n### 從 DenseNet-201 一路改過來\n\n作者的做法和 ConvNeXt 很像，從一個舊模型出發，一次改一個地方，記下每一步的準確率和速度。起點是 DenseNet-201，先換上現代的訓練配方，AdamW、cosine 學習率、300 個 epoch、各種資料增強，ImageNet 是 79.7%。\n\n| 步驟 | top-1 | 參數量 | FLOPs | 推論延遲（批次 1） | 訓練延遲 | 訓練記憶體 |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| DenseNet-201（新配方） | 79.7% | 20.0M | 4.3G | 38.4 ms | 131 ms | 3.9 GB |\n| 加寬、變淺 | 79.5% | 21.8M | 11.1G | 8.5 ms | 85 ms | 3.2 GB |\n| 換成現代的區塊 | 80.4% | 12.9M | 4.8G | 10.4 ms | 112 ms | 3.4 GB |\n| 擴張倍數改跟著輸入走 | 80.8% | 19.9M | 4.7G | 11.8 ms | 88 ms | 3.1 GB |\n| 更多過渡層、各階段不同 GR | 82.3% | 21.2M | 5.0G | 11.0 ms | 90 ms | 3.4 GB |\n| patchify stem | 82.4% | 21.2M | 4.9G | 11.0 ms | 88 ms | 3.2 GB |\n| 重新設計過渡層 | 82.6% | 22.4M | 4.9G | 13.6 ms | 97 ms | 3.1 GB |\n| 通道重新縮放 | **82.8%** | 23.9M | 5.0G | 14.0 ms | 99 ms | 3.1 GB |\n\n*表：摘自原論文 Table 1 的部分欄位。*\n\n一步一步看。\n\n**加寬、變淺。** GR 從 32 拉到 120，四個階段的區塊數從 $(6, 12, 48, 32)$ 砍到 $(3, 3, 12, 3)$。準確率幾乎沒變，訓練延遲少了 35%，記憶體少了 18%。FLOPs 漲到 11.1G，後面再處理。\n\n**換成現代的區塊。** 每個區塊改成 ConvNeXt 的樣子，7×7 depthwise 卷積、LayerNorm、1×1 卷積擴張再壓回來，正規化和啟動函數也減少。唯一的差別是輸出通道是 GR，比輸入的通道數少，每個區塊都在把前面接起來的特徵壓縮成 GR 個新通道。這一步多了 0.9 個百分點。\n\n**擴張倍數跟著輸入走。** 原版 DenseNet 的瓶頸層寬度是 GR 的 4 倍。作者改成輸入通道數的 4 倍，讓非線性之前的特徵更豐富，再把 GR 從 120 減半到 60，控制計算量。準確率多 0.4，訓練還快了 21%。\n\n**更多過渡層。** 原版 DenseNet 只在階段之間放過渡層，順便降採樣。作者在階段裡面每 3 個區塊就放一個步長 1 的過渡層，只負責把通道壓成一半。省下來的計算量拿去把 GR 調高，各階段的 GR 也不再一樣。這一步多了 1.5 個百分點，是整個流程裡最大的一步。\n\n最後三步比較小。stem 改成 4×4、步長 4 的 patchify；過渡層拿掉平均池化，直接用步長等於卷積核大小的卷積降採樣；再加上通道重新縮放（layer scale 加上 effective SE），各多 0.1 到 0.2。\n\n[![原論文 Fig. 1：上半部左邊是 RDNet 的整體流程，依序是 Stem、Stage-1、Transition S/2、Stage-2、Transition S/2、Stage-3、Transition S/2、Stage-4、GlobalAvgPool、LayerNorm、FC；右邊是 Feature mixer f，依序是 DW Conv 7×7（C→C）、LayerNorm、Linear（C→4C）、GeLU、Linear（4C→GR）。下半部左邊展開 Stage-N，紫色的特徵方塊一路串接變厚，每經過一個 f 就多一片，最後接一個 Transition S/1，整組重複 L_N 次；右邊是 Transition layer S/s，由 LayerNorm 和步長 s 的 s×s 卷積組成，通道數減半。](/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png)](/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png)\n\n*原論文 Fig. 1（Kim 等人），取自 [**PDF 第 6 頁**](https://arxiv.org/pdf/2403.19588v2#page=6)。圖說未收錄。*\n\n最後的 RDNet 長這樣。下半部是一個階段的內部，紫色方塊每經過一個特徵混合器 $f$ 就多一片，三個 $f$ 之後接一個步長 1 的過渡層，把厚度壓回一半，這樣的組合重複 $L_N$ 次。\n\n### 跟著通道數走一遍\n\n光看圖不容易感覺到通道數怎麼變化。timm 裡有 RDNet 的實作和 `rdnet_tiny` 到 `rdnet_large` 的預訓練權重，我們直接建一個 RDNet-T（不下載權重），印出每組區塊的通道數：\n\n```python\nimport timm\nimport torch.nn as nn\n\nmodel = timm.create_model(\"rdnet_tiny\", pretrained=False)    # 只要架構，不下載權重\nprint(\"stem 輸出\", model.stem[0].out_channels, \"個通道\")\nfor i, group in enumerate(model.dense_stages):\n    *transition, dense = group.children()\n    if transition:                                            # 組與組之間的過渡層：LayerNorm + 卷積\n        conv = transition[1]\n        kind = \"降採樣\" if conv.stride[0] == 2 else \"只壓通道\"\n        print(f\"  過渡層（{kind}） {conv.in_channels} → {conv.out_channels}\")\n    blocks = list(dense.children())\n    c_in = blocks[0].layers.layers[0].in_channels             # 第一個區塊的 7×7 depthwise 吃進多少通道\n    gr = blocks[0].growth_rate\n    print(f\"第 {i} 組 {len(blocks)} 個區塊，每塊接上 {gr} 個通道  {c_in} → {c_in + len(blocks) * gr}\")\n```\n\n```text\nstem 輸出 64 個通道\n第 0 組 3 個區塊，每塊接上 64 個通道  64 → 256\n  過渡層（降採樣） 256 → 128\n第 1 組 3 個區塊，每塊接上 104 個通道  128 → 440\n  過渡層（降採樣） 440 → 216\n第 2 組 3 個區塊，每塊接上 128 個通道  216 → 600\n  過渡層（只壓通道） 600 → 296\n第 3 組 3 個區塊，每塊接上 128 個通道  296 → 680\n  過渡層（只壓通道） 680 → 336\n第 4 組 3 個區塊，每塊接上 128 個通道  336 → 720\n  過渡層（只壓通道） 720 → 360\n第 5 組 3 個區塊，每塊接上 128 個通道  360 → 744\n  過渡層（降採樣） 744 → 368\n第 6 組 3 個區塊，每塊接上 224 個通道  368 → 1040\n```\n\ntimm 把每 3 個區塊包成一組，第 2 到第 5 組合起來就是論文說的第三階段，12 個區塊。每組結束時通道變多，過渡層再壓回一半，所以第三階段的寬度一直在 216 到 744 之間來回。\n\n如果第三階段中間不放過渡層呢？我們用同樣的數字估一下。每個區塊最貴的是兩個 1×1 卷積，先擴張到輸入的 4 倍，再壓回 GR，這兩個卷積每個位置的乘加數大約是 $4c^2 + 4c\\cdot\\mathrm{GR}$：\n\n```python\ndef stage3(every, c=216, gr=128, blocks=12):\n    macs = 0\n    for b in range(blocks):\n        if b and b % every == 0:\n            c = int(c * 0.5 / 8) * 8                          # 過渡層把通道壓成一半\n        macs += c * 4 * c + 4 * c * gr                        # 每個位置：1×1 擴張到 4 倍，再壓回 GR\n        c += gr                                               # 輸出接到後面\n    return macs, c\n\nfor every, label in ((3, \"每 3 塊壓一次\"), (6, \"每 6 塊壓一次\"), (12, \"整段都不壓\")):\n    macs, c = stage3(every)\n    print(f\"{label}：最後 {c} 個通道，兩個 1×1 卷積每個位置 {macs / 1e6:.1f}M 次乘加\")\n```\n\n```text\n每 3 塊壓一次：最後 744 個通道，兩個 1×1 卷積每個位置 12.2M 次乘加\n每 6 塊壓一次：最後 1256 個通道，兩個 1×1 卷積每個位置 29.0M 次乘加\n整段都不壓：最後 1752 個通道，兩個 1×1 卷積每個位置 55.7M 次乘加\n```\n\n這裡省略了 depthwise 卷積和 SE，只算最大頭的部分。不放過渡層，第三階段最後會長到 1752 個通道，計算量是每 3 塊壓一次的 4.6 倍。因為擴張層的成本和通道數的平方成正比，串接讓通道線性成長，計算量就跟著平方成長。頻繁地壓，才有本錢把 GR 調高。\n\n論文的消融也支持這個間隔。在差不多的計算量下，每 2 塊壓一次是 82.7%，每 3 塊 82.8%，每 4 塊 82.6%，每 6 塊掉到 82.1%。\n\n## 討論\n\n### ImageNet\n\n先和幾個代表性的模型比，全部只用 ImageNet-1K 訓練，延遲是一次推論 128 張圖，記憶體用批次 16 量：\n\n| 模型 | 參數量 | FLOPs | 延遲（ms） | 記憶體（GB） | top-1 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| Swin-T | 28M | 4.5G | 173 | 2.6 | 81.3% |\n| ConvNeXt-T | 29M | 4.5G | 150 | 2.7 | 82.1% |\n| InceptionNeXt-T | 28M | 4.2G | 132 | 3.3 | 82.3% |\n| RDNet-T | 24M | 5.0G | 175 | 4.1 | **82.8%** |\n| ConvNeXt-S | 50M | 8.7G | 266 | 4.0 | 83.1% |\n| RDNet-S | 50M | 8.7G | 289 | 5.4 | **83.7%** |\n| ConvNeXt-B | 89M | 15.4G | 417 | 5.4 | 83.8% |\n| RDNet-B | 87M | 15.4G | 472 | 6.9 | **84.4%** |\n\n*表：摘自原論文 Table 4 的部分列。*\n\nRDNet 每個尺寸都比 ConvNeXt 多 0.6 到 0.7 個百分點，代價是推論延遲多了大約一到兩成，記憶體也多了大約三到五成。串接要保留前面所有的特徵，記憶體還是比較吃。\n\n作者另外拿 2022 到 2024 年的新模型來比，HorNet、VAN、BiFormer、NAT、SMT、MogaNet。這時情況反過來，RDNet-T 和它們差不多大，準確率持平或低了最多 1 個百分點，批次 1 的延遲卻只有 7.4 ms，其他模型在 20 到 52 ms 之間。作者在表的說明裡也直接寫了，他們的模型準確率稍微落後，靠速度補回來。\n\n### 下游任務\n\n分割和偵測的優勢比較明顯。ADE20K 上接 UperNet，RDNet-T 的單尺度 mIoU 是 47.6，ConvNeXt-T 是 46.0；B 是 49.6 對 49.1。COCO 上用 Mask R-CNN 訓練 3 倍的排程，RDNet-T 的 box AP 是 47.5，ConvNeXt-T 是 46.2，參數還少了 5M。作者認為，串接保留了不同層的特徵，這對密集預測特別有用。\n\n作者也用 OpenCLIP 訓練了 CLIP，影像編碼器分別用 RDNet-B 和 ConvNeXt-B，ImageNet 零樣本分類的 top-1 是 54.1% 對 51.2%。\n\n### stochastic depth 也有用\n\n原版 DenseNet 沒有用 stochastic depth，作者說是因為兩者的連接模式本來就有相似之處。實際試了之後，影響很大，比例 0 是 81.6%，0.15 是 82.8%，多了 1.2 個百分點。\n\n\n> [!TIP]\n> \n> stochastic depth 是怎麼隨機讓一層「請假」的，可以看這篇：\n> \n> - **[[16.03] Stochastic Depth：今天這層請假](/blog/papers/classic-cnns/stochastic-depth)**\n\n\n## 結論\n\n最後整理一下：\n\n1. **串接本身**：一萬五千個隨機網路裡，串接平均比加法高 0.7 到 1.7 個百分點，記憶體也多了一到兩成。\n2. **讓 DenseNet 能打的是身材**：加寬變淺讓它變快，更多過渡層讓它能把 GR 調高，後者一步就多了 1.5 個百分點。\n3. **成績**：同樣大小比 ConvNeXt 準 0.6 到 0.7 個百分點，分割和偵測的差距更大；推論慢一些，記憶體多三到五成。\n\n這樣看來，DenseNet 當年被冷落，問題大多出在又深又窄的身材和舊的訓練配方，串接本身倒是沒做錯什麼。只是它要的記憶體，從 2016 年到現在都沒少過。\n","imageSizes":{"/content/papers/assets/47b1a145d10885d762e8f5476e49d96dc1e35fdc1e3227a66162ec1bf6870710.png":{"width":1230,"height":615}}}