{"notes":[{"id":"classic-cnns/resnext","category":"classic-cnns","legacyPath":"classic-cnns/1611-resnext","legacySlug":"1611-resnext","legacyPaths":[],"title":"ResNeXt","subtitle":"探訪下一個維度","author":"Z. Yuan","paperDate":"2016-11-16","publishedOn":"2024-07-12","paperTitle":"Aggregated Residual Transformations for Deep Neural Networks","paperUrl":"https://arxiv.org/abs/1611.05431","bibliography":{"title":"Aggregated Residual Transformations for Deep Neural Networks","authors":["Saining Xie","Ross Girshick","Piotr Dollár","Zhuowen Tu","Kaiming He"],"url":"https://arxiv.org/abs/1611.05431","pdfUrl":"https://arxiv.org/pdf/1611.05431","codeUrl":"https://github.com/facebookresearch/ResNeXt","preprint":{"id":"1611.05431","submittedOn":"2016-11-16","url":"https://arxiv.org/abs/1611.05431"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1611.05431"},"citation":"@misc{classiccnnsresnext2016,\n  title = {{Aggregated Residual Transformations for Deep Neural Networks}},\n  author = {Saining Xie and Ross Girshick and Piotr Dollár and Zhuowen Tu and Kaiming He},\n  year = {2016},\n  eprint = {1611.05431},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1611.05431}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnext.4945bb6b3dae.bib"},"thumbnail":{"src":"/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg","width":1224,"height":576},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/resnext.6f895ad74065.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"下游任務泛化性","text":"下游任務泛化性","depth":3},{"id":"不利於模型設計","text":"不利於模型設計","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"網路架構","text":"網路架構","depth":3},{"id":"模型配置","text":"模型配置","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-1k-訓練效果","text":"ImageNet-1K 訓練效果","depth":3},{"id":"增加模型複雜度","text":"增加模型複雜度","depth":3},{"id":"imagenet-1k-模型表現","text":"ImageNet-1K 模型表現","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分組卷積與特徵重用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分組卷積與特徵重用"}]},{"id":"classic-cnns/inception-v1","category":"classic-cnns","legacyPath":"classic-cnns/1409-inception-v1","legacySlug":"1409-inception-v1","legacyPaths":[],"title":"GoogLeNet","subtitle":"或叫做 Inception-V1","author":"Z. Yuan","paperDate":"2014-09-17","publishedOn":"2024-09-22","paperTitle":"Going Deeper with Convolutions","paperUrl":"https://arxiv.org/abs/1409.4842","bibliography":{"title":"Going Deeper with Convolutions","authors":["Christian Szegedy","Wei Liu","Yangqing Jia","Pierre Sermanet","Scott Reed","Dragomir Anguelov","Dumitru Erhan","Vincent Vanhoucke","Andrew Rabinovich"],"url":"https://arxiv.org/abs/1409.4842","pdfUrl":"https://arxiv.org/pdf/1409.4842","codeUrl":null,"preprint":{"id":"1409.4842","submittedOn":"2014-09-17","url":"https://arxiv.org/abs/1409.4842"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2015.7298594"},"citation":"@misc{classiccnnsinceptionv12014,\n  title = {{Going Deeper with Convolutions}},\n  author = {Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich},\n  year = {2014},\n  eprint = {1409.4842},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1409.4842}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v1.3d20e0e32bcd.bib"},"thumbnail":{"src":"/content/papers/assets/254b970b1374c24267b60c2d684338760e9ba8b10dad761a5fa15f1bb8bcf6ef.jpg","width":1330,"height":394},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/inception-v1.91efdf81ddcf.md","headings":[{"id":"解決問題","text":"解決問題","depth":2},{"id":"inception-模塊","text":"Inception 模塊","depth":3},{"id":"參數配置","text":"參數配置","depth":3},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"在-ilsvrc-2014-上的表現","text":"在 ILSVRC 2014 上的表現","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分組卷積與特徵重用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分組卷積與特徵重用"}]},{"id":"classic-cnns/cspnet","category":"classic-cnns","legacyPath":"classic-cnns/1911-cspnet","legacySlug":"1911-cspnet","legacyPaths":[],"title":"CSPNet","subtitle":"半半半插件","author":"Z. Yuan","paperDate":"2019-11-27","publishedOn":"2024-09-29","paperTitle":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","paperUrl":"https://arxiv.org/abs/1911.11929","bibliography":{"title":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","authors":["Chien-Yao Wang","Hong-Yuan Mark Liao","I-Hau Yeh","Yueh-Hua Wu","Ping-Yang Chen","Jun-Wei Hsieh"],"url":"https://arxiv.org/abs/1911.11929","pdfUrl":"https://arxiv.org/pdf/1911.11929","codeUrl":null,"preprint":{"id":"1911.11929","submittedOn":"2019-11-27","url":"https://arxiv.org/abs/1911.11929"},"publication":{"venue":"CVPR Workshops","year":2020,"kind":"workshop","url":"https://doi.org/10.1109/cvprw50498.2020.00203"},"citation":"@misc{classiccnnscspnet2019,\n  title = {{CSPNet: A New Backbone that can Enhance Learning Capability of CNN}},\n  author = {Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},\n  year = {2019},\n  eprint = {1911.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1911.11929}\n}\n","citationPath":"/content/papers/citations/classic-cnns/cspnet.2914b9492a0c.bib"},"thumbnail":{"src":"/content/papers/assets/2602af8b5f92cba820e545c84abd94943cab1fe2631cf83834972b54b374ed20.jpg","width":2809,"height":1911},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/cspnet.df77c5a48e8b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"transition-layer","text":"Transition Layer","depth":3},{"id":"exact-fusion-model","text":"Exact Fusion Model","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"csp-架構的有效性","text":"CSP 架構的有效性","depth":3},{"id":"efm-架構的有效性","text":"EFM 架構的有效性","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"ms-coco-上的性能","text":"MS COCO 上的性能","depth":3},{"id":"推論速度","text":"推論速度","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分組卷積與特徵重用"},{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分組卷積與特徵重用"}]}],"text":"還記得稍早提出的 Inception 模型嗎？\n\n- [**Going Deeper with Convolutions**](https://arxiv.org/abs/1409.4842)\n\n那讓人眼花撩亂的分支結構，是不是讓你印象深刻？\n\n## 定義問題\n\nInception 證明了精心設計的拓墣結構可以用較低的理論複雜度達到更好的性能。\n\n但這種設計方式會帶來幾個問題：\n\n### 下游任務泛化性\n\n在 Inception 模型中，通常會使用不同大小的卷積核（例如 1x1、3x3、5x5 等）來同時進行不同尺度的特徵提取。\n\n我們需要根據數據集特性來確定每一層使用的卷積核大小和數量。例如，對於圖片數據集，較大的卷積核可以捕捉到更多的上下文信息，而較小的卷積核可以提取到更精細的特徵。\n\n每個 Inception 模塊通常包括多個不同大小的卷積層和池化層，根據需求設計每個模塊內部的結構，並使用交叉驗證來選擇效果最好的組合。\n\n\n> [!TIP]\n> \n> 這不符合人體工學啊！\n> \n> 每當你要拿這個模型來用，你就開始躊躇......根本是反工程師的集大成。\n\n\n### 不利於模型設計\n\n目前對於神經網路的研究，已經從「特徵工程」漸漸地轉向「模型設計工程」，即如何設計一個好的神經網路。\n\nInception 模型的設計需要大量的人力和時間，並且不易擴展到更大的模型。\n\n但是這個「分割-變化-合併」的設計理念，還是非常值得借鑒的。\n\n## 解決問題\n\n### 網路架構\n\n![img2](/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg)\n\n如上圖所示，左邊是原始的 ResNet 結構，右邊是作者提出的 ResNeXt 結構。\n\n作者借鑑了 VGG 和 ResNet 的設計方法，採用高度模組化的設計，使得模型由一堆殘差塊（residual blocks）組成。接著引入了一個新的維度：基數（cardinality），作為除了深度和寬度之外的重要因素。\n\n這個 cardinality 指的是在 ResNeXt 模塊中的分組數量，即：將輸入通道分為幾組進行卷積運算。例如上圖中的 `total 32 paths` 中的 `32` 就是分組數，一組有四個通道。\n\n再來，作者通過兩個簡單規則來簡化設計空間，使得整個網路的設計變得更加簡單和統一：\n\n- 規則一：如果生成相同大小的空間圖，則這些區塊共享相同的超參數（寬度和過濾器大小）。\n- 規則二：每次將空間圖下採樣 2 倍時，模組的寬度乘以 2 倍。\n\n最後是聚合變換（Aggregated Transformations）的概念，將神經元的基本操作擴展到更複雜的變換上，並且使用瓶頸型架構和分組卷積的方法來簡化實現和提高模型性能。\n\n這個架構存在幾種等價形式，如下圖：\n\n![img4](/content/papers/assets/28910e4d22dc8f2c055dc119b8fbd9728731c45a041ef67bacc24c4ad7e6301c.jpg)\n\n- 上圖 (a) 是本論文提出的結構。\n- 上圖 (b) 是先串接各分支 3×3 卷積的輸出，再共用一個 128→256 的 1×1 卷積。\n- 上圖 (c) 是用 32 組的 3×3 分組卷積實現這些分支，最後同樣接 128→256 的 1×1 卷積。\n\n\n> [!TIP]\n> \n> 在撇開 Norm、ReLU 和 Pooling 的情況下，這幾個架構的輸出是等價的。\n> \n> 我們在實作的時候會直接用第三種方式，快速又簡單。\n\n\n### 模型配置\n\n![img3](/content/papers/assets/e30a91af00c174fae7a49cc498f42a513476128d1f427ba921c50eb7d595d9c2.jpg)\n\n使用分組卷積會降低運算量，因此為了維持相同的比較基準，作者將 ResNeXt 模組中的中間通道數量增加為原本 ResNet 的 2 倍，然後取分組數為 32。\n\n## 討論\n\n### ImageNet-1K 訓練效果\n\n![img1](/content/papers/assets/f1b9572741682865800ae1649291b9939705a8639603bcaa4d99e3cb547a3feb.jpg)\n\n上圖展示了不同模型的驗證誤差和訓練誤差隨著訓練次數（epochs）的變化情況。\n\n與 ResNet-50 相比，32×4d ResNeXt-50 的 top-1 驗證錯誤率為 22.2%，比 ResNet-50 的 23.9% 低了 1.7 個百分點。\n\n隨著基數 $C$ 從 1 增加到 32，同時保持模型複雜性不變，錯誤率不斷降低。32×4d ResNeXt 的訓練誤差也明顯低於 ResNet-50，表明性能提升主要來自於模型表示能力的增強，而非正則化效應。\n\n在 ResNet-101 的情況下，32×4d ResNeXt-101 的 top-1 驗證錯誤率比 ResNet-101 低 0.8 個百分點。儘管驗證誤差的改善比 ResNet-50 的情況小，但訓練誤差的改善仍然顯著（ResNet-101 為 20%，32×4d ResNeXt-101 為 16%）。\n\n### 增加模型複雜度\n\n![img6](/content/papers/assets/8c576fcc821cb0a29fa73df0db77e811a3ff980ee24848c132c2f5ab38550dc5.jpg)\n\n為了測試分組數量對模型性能的影響，作者進行了以下幾個實驗：\n\n1. **將複雜性增加兩倍**\n\n   - 與 ResNet-101 基線 (22.0%) 相比，增加模型複雜性總能減少錯誤。\n   - 但效果有限：\n     - 增加深度（ResNet-200，降低 0.3 個百分點）\n     - 增加寬度（更寬的 ResNet-101，降低 0.7 個百分點）。\n\n2. **增加基數的效果更好**\n\n   實驗顯示增加基數 $C$ 比增加深度或寬度能顯示出更好的結果。\n\n   - 2×64d ResNeXt-101 將 top-1 誤差降低至 20.7%。\n     - 在 1×64d ResNet-101 基線上加倍 $C$ 並保持寬度\n   - 64×4d ResNeXt-101 將 top-1 誤差降低至 20.4%。\n     - 在 32×4d ResNeXt-101 上加倍 $C$ 並保持寬度\n\n3. **較低複雜度下的性能**\n\n    32×4d ResNeXt-101 (21.2%) 的性能優於更深的 ResNet-200 和更寬的 ResNet-101，儘管其複雜度只有約 50%。這表明基數 $C$ 是比深度和寬度更有效的維度。\n\n### ImageNet-1K 模型表現\n\n![img5](/content/papers/assets/519d441ad166763fc8b8b33bd4d5850063e7ab1ed919663c3d53af746f07bd6e.jpg)\n\n上表顯示了在 ImageNet 驗證集上進行單次裁切測試的結果。\n\nResNeXt 的結果與 ResNet、Inception-v3/v4 和 Inception-ResNet-v2 相比毫不遜色；ResNeXt-101 (64×4d) 以 320×320 單次裁切達到 4.4% 的 top-5 錯誤率。\n\n不僅如此，ResNeXt 的架構設計比所有 Inception 模型簡單得多，手動設定的超參數也少得多。\n\n## 結論\n\n本論文提出了一種簡單且高度模組化的網路架構 ResNeXt，通過重複聚合一組具有相同拓撲結構的變換構建而成，形成了一種同質的多分支架構，只需設定少量的超參數。\n\n作者引入了一個新的維度：基數（cardinality），作為除了深度和寬度之外的重要因素。\n\n實驗結果表明，在保持模型複雜度的條件下，增加基數能夠顯著提高分類準確率。\n\n相比於單純增加模型的深度或寬度，增加基數更為有效。\n\n\n> [!TIP]\n> \n> 這份論文為之後的可分離卷積、MobileNet 等輕量化模型的發展提供了重要的參考。\n","imageSizes":{"/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg":{"width":1224,"height":576},"/content/papers/assets/28910e4d22dc8f2c055dc119b8fbd9728731c45a041ef67bacc24c4ad7e6301c.jpg":{"width":1632,"height":506},"/content/papers/assets/e30a91af00c174fae7a49cc498f42a513476128d1f427ba921c50eb7d595d9c2.jpg":{"width":1044,"height":1024},"/content/papers/assets/f1b9572741682865800ae1649291b9939705a8639603bcaa4d99e3cb547a3feb.jpg":{"width":1356,"height":580},"/content/papers/assets/8c576fcc821cb0a29fa73df0db77e811a3ff980ee24848c132c2f5ab38550dc5.jpg":{"width":1110,"height":594},"/content/papers/assets/519d441ad166763fc8b8b33bd4d5850063e7ab1ed919663c3d53af746f07bd6e.jpg":{"width":1208,"height":544}}}
