{"notes":[{"id":"classic-cnns/resnext","category":"classic-cnns","legacyPath":"classic-cnns/1611-resnext","legacySlug":"1611-resnext","legacyPaths":[],"title":"ResNeXt","subtitle":"探访下一个维度","author":"Z. Yuan","paperDate":"2016-11-16","publishedOn":"2024-07-12","paperTitle":"Aggregated Residual Transformations for Deep Neural Networks","paperUrl":"https://arxiv.org/abs/1611.05431","bibliography":{"title":"Aggregated Residual Transformations for Deep Neural Networks","authors":["Saining Xie","Ross Girshick","Piotr Dollár","Zhuowen Tu","Kaiming He"],"url":"https://arxiv.org/abs/1611.05431","pdfUrl":"https://arxiv.org/pdf/1611.05431","codeUrl":"https://github.com/facebookresearch/ResNeXt","preprint":{"id":"1611.05431","submittedOn":"2016-11-16","url":"https://arxiv.org/abs/1611.05431"},"publication":{"venue":"CVPR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1611.05431"},"citation":"@misc{classiccnnsresnext2016,\n  title = {{Aggregated Residual Transformations for Deep Neural Networks}},\n  author = {Saining Xie and Ross Girshick and Piotr Dollár and Zhuowen Tu and Kaiming He},\n  year = {2016},\n  eprint = {1611.05431},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1611.05431}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnext.4945bb6b3dae.bib"},"thumbnail":{"src":"/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg","width":1224,"height":576},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/resnext.b391d1bf5dbf.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"下游任务泛化性","text":"下游任务泛化性","depth":3},{"id":"不利于模型设计","text":"不利于模型设计","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"网络架构","text":"网络架构","depth":3},{"id":"模型配置","text":"模型配置","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet-1k-训练效果","text":"ImageNet-1K 训练效果","depth":3},{"id":"增加模型复杂度","text":"增加模型复杂度","depth":3},{"id":"imagenet-1k-模型表现","text":"ImageNet-1K 模型表现","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分组卷积与特征复用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分组卷积与特征复用"}]},{"id":"classic-cnns/inception-v1","category":"classic-cnns","legacyPath":"classic-cnns/1409-inception-v1","legacySlug":"1409-inception-v1","legacyPaths":[],"title":"GoogLeNet","subtitle":"或叫做 Inception-V1","author":"Z. Yuan","paperDate":"2014-09-17","publishedOn":"2024-09-22","paperTitle":"Going Deeper with Convolutions","paperUrl":"https://arxiv.org/abs/1409.4842","bibliography":{"title":"Going Deeper with Convolutions","authors":["Christian Szegedy","Wei Liu","Yangqing Jia","Pierre Sermanet","Scott Reed","Dragomir Anguelov","Dumitru Erhan","Vincent Vanhoucke","Andrew Rabinovich"],"url":"https://arxiv.org/abs/1409.4842","pdfUrl":"https://arxiv.org/pdf/1409.4842","codeUrl":null,"preprint":{"id":"1409.4842","submittedOn":"2014-09-17","url":"https://arxiv.org/abs/1409.4842"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2015.7298594"},"citation":"@misc{classiccnnsinceptionv12014,\n  title = {{Going Deeper with Convolutions}},\n  author = {Christian Szegedy and Wei Liu and Yangqing Jia and Pierre Sermanet and Scott Reed and Dragomir Anguelov and Dumitru Erhan and Vincent Vanhoucke and Andrew Rabinovich},\n  year = {2014},\n  eprint = {1409.4842},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1409.4842}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v1.3d20e0e32bcd.bib"},"thumbnail":{"src":"/content/papers/assets/254b970b1374c24267b60c2d684338760e9ba8b10dad761a5fa15f1bb8bcf6ef.jpg","width":1330,"height":394},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/inception-v1.7c8c2ff19589.md","headings":[{"id":"解决问题","text":"解决问题","depth":2},{"id":"inception-模块","text":"Inception 模块","depth":3},{"id":"参数配置","text":"参数配置","depth":3},{"id":"模型架构","text":"模型架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"在-ilsvrc-2014-上的表现","text":"在 ILSVRC 2014 上的表现","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分组卷积与特征复用"},{"id":"classic-cnns/cspnet","relation":"comparison","topic":"分支、分组卷积与特征复用"}]},{"id":"classic-cnns/cspnet","category":"classic-cnns","legacyPath":"classic-cnns/1911-cspnet","legacySlug":"1911-cspnet","legacyPaths":[],"title":"CSPNet","subtitle":"半半半插件","author":"Z. Yuan","paperDate":"2019-11-27","publishedOn":"2024-09-29","paperTitle":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","paperUrl":"https://arxiv.org/abs/1911.11929","bibliography":{"title":"CSPNet: A New Backbone that can Enhance Learning Capability of CNN","authors":["Chien-Yao Wang","Hong-Yuan Mark Liao","I-Hau Yeh","Yueh-Hua Wu","Ping-Yang Chen","Jun-Wei Hsieh"],"url":"https://arxiv.org/abs/1911.11929","pdfUrl":"https://arxiv.org/pdf/1911.11929","codeUrl":null,"preprint":{"id":"1911.11929","submittedOn":"2019-11-27","url":"https://arxiv.org/abs/1911.11929"},"publication":{"venue":"CVPR Workshops","year":2020,"kind":"workshop","url":"https://doi.org/10.1109/cvprw50498.2020.00203"},"citation":"@misc{classiccnnscspnet2019,\n  title = {{CSPNet: A New Backbone that can Enhance Learning Capability of CNN}},\n  author = {Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},\n  year = {2019},\n  eprint = {1911.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1911.11929}\n}\n","citationPath":"/content/papers/citations/classic-cnns/cspnet.2914b9492a0c.bib"},"thumbnail":{"src":"/content/papers/assets/2602af8b5f92cba820e545c84abd94943cab1fe2631cf83834972b54b374ed20.jpg","width":2809,"height":1911},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/cspnet.49f7ed2a19fe.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型架构","text":"模型架构","depth":3},{"id":"transition-layer","text":"Transition Layer","depth":3},{"id":"exact-fusion-model","text":"Exact Fusion Model","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"csp-架构的有效性","text":"CSP 架构的有效性","depth":3},{"id":"efm-架构的有效性","text":"EFM 架构的有效性","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"ms-coco-上的性能","text":"MS COCO 上的性能","depth":3},{"id":"推理速度","text":"推理速度","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"comparison","topic":"分支、分组卷积与特征复用"},{"id":"classic-cnns/resnext","relation":"comparison","topic":"分支、分组卷积与特征复用"}]}],"text":"还记得稍早提出的 Inception 模型吗？\n\n- [**Going Deeper with Convolutions**](https://arxiv.org/abs/1409.4842)\n\n那让人眼花缭乱的分支结构，是不是让你印象深刻？\n\n## 定义问题\n\nInception 证明了精心设计的拓扑结构可以用较低的理论复杂度达到更好的性能。\n\n但这种设计方式会带来几个问题：\n\n### 下游任务泛化性\n\n在 Inception 模型中，通常会使用不同大小的卷积核（例如 1x1、3x3、5x5 等）来同时进行不同尺度的特征提取。\n\n我们需要根据数据集特性来确定每一层使用的卷积核大小和数量。例如，对于图片数据集，较大的卷积核可以捕捉到更多的上下文信息，而较小的卷积核可以提取到更精细的特征。\n\n每个 Inception 模块通常包括多个不同大小的卷积层和池化层，根据需求设计每个模块内部的结构，并使用交叉验证来选择效果最好的组合。\n\n\n> [!TIP]\n> \n> 这不符合人体工学啊！\n> \n> 每当你要拿这个模型来用，你就开始踌躇......根本是反工程师的集大成。\n\n\n### 不利于模型设计\n\n目前对于神经网络的研究，已经从「特征工程」渐渐地转向「模型设计工程」，即如何设计一个好的神经网络。\n\nInception 模型的设计需要大量的人力和时间，并且不易扩展到更大的模型。\n\n但是这个「分割-变化-合并」的设计理念，还是非常值得借鉴的。\n\n## 解决问题\n\n### 网络架构\n\n![img2](/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg)\n\n如上图所示，左边是原始的 ResNet 结构，右边是作者提出的 ResNeXt 结构。\n\n作者借鉴了 VGG 和 ResNet 的设计方法，采用高度模块化的设计，使得模型由一堆残差块（residual blocks）组成。接着引入了一个新的维度：基数（cardinality），作为除了深度和宽度之外的重要因素。\n\n这个 cardinality 指的是在 ResNeXt 模块中的分组数量，即：将输入通道分为几组进行卷积运算。例如上图中的 `total 32 paths` 中的 `32` 就是分组数，一组有四个通道。\n\n再来，作者通过两个简单规则来简化设计空间，使得整个网络的设计变得更加简单和统一：\n\n- 规则一：如果生成相同大小的空间图，则这些区块共享相同的超参数（宽度和滤波器大小）。\n- 规则二：每次将空间图下采样 2 倍时，模块的宽度乘以 2 倍。\n\n最后是聚合变换（Aggregated Transformations）的概念，将神经元的基本操作扩展到更复杂的变换上，并且使用瓶颈型架构和分组卷积的方法来简化实现和提高模型性能。\n\n这个架构存在几种等价形式，如下图：\n\n![img4](/content/papers/assets/28910e4d22dc8f2c055dc119b8fbd9728731c45a041ef67bacc24c4ad7e6301c.jpg)\n\n- 上图 (a) 是本论文提出的结构。\n- 上图 (b) 是先串接各分支 3×3 卷积的输出，再共用一个 128→256 的 1×1 卷积。\n- 上图 (c) 是用 32 组的 3×3 分组卷积实现这些分支，最后同样接 128→256 的 1×1 卷积。\n\n\n> [!TIP]\n> \n> 在撇开 Norm、ReLU 和 Pooling 的情况下，这几个架构的输出是等价的。\n> \n> 我们在实现的时候会直接用第三种方式，快速又简单。\n\n\n### 模型配置\n\n![img3](/content/papers/assets/e30a91af00c174fae7a49cc498f42a513476128d1f427ba921c50eb7d595d9c2.jpg)\n\n使用分组卷积会降低运算量，因此为了维持相同的比较基准，作者将 ResNeXt 模块中的中间通道数量增加为原本 ResNet 的 2 倍，然后取分组数为 32。\n\n## 讨论\n\n### ImageNet-1K 训练效果\n\n![img1](/content/papers/assets/f1b9572741682865800ae1649291b9939705a8639603bcaa4d99e3cb547a3feb.jpg)\n\n上图展示了不同模型的验证误差和训练误差随着训练次数（epochs）的变化情况。\n\n与 ResNet-50 相比，32×4d ResNeXt-50 的 top-1 验证错误率为 22.2%，比 ResNet-50 的 23.9% 低了 1.7 个百分点。\n\n随着基数 $C$ 从 1 增加到 32，同时保持模型复杂性不变，错误率不断降低。32×4d ResNeXt 的训练误差也明显低于 ResNet-50，表明性能提升主要来自于模型表示能力的增强，而非正则化效应。\n\n在 ResNet-101 的情况下，32×4d ResNeXt-101 的 top-1 验证错误率比 ResNet-101 低 0.8 个百分点。尽管验证误差的改善比 ResNet-50 的情况小，但训练误差的改善仍然显著（ResNet-101 为 20%，32×4d ResNeXt-101 为 16%）。\n\n### 增加模型复杂度\n\n![img6](/content/papers/assets/8c576fcc821cb0a29fa73df0db77e811a3ff980ee24848c132c2f5ab38550dc5.jpg)\n\n为了测试分组数量对模型性能的影响，作者进行了以下几个实验：\n\n1. **将复杂性增加两倍**\n\n   - 与 ResNet-101 基线 (22.0%) 相比，增加模型复杂性总能减少错误。\n   - 但效果有限：\n     - 增加深度（ResNet-200，降低 0.3 个百分点）\n     - 增加宽度（更宽的 ResNet-101，降低 0.7 个百分点）。\n\n2. **增加基数的效果更好**\n\n   实验显示增加基数 $C$ 比增加深度或宽度能显示出更好的结果。\n\n   - 2×64d ResNeXt-101 将 top-1 误差降低至 20.7%。\n     - 在 1×64d ResNet-101 基线上加倍 $C$ 并保持宽度\n   - 64×4d ResNeXt-101 将 top-1 误差降低至 20.4%。\n     - 在 32×4d ResNeXt-101 上加倍 $C$ 并保持宽度\n\n3. **较低复杂度下的性能**\n\n    32×4d ResNeXt-101 (21.2%) 的性能优于更深的 ResNet-200 和更宽的 ResNet-101，尽管其复杂度只有约 50%。这表明基数 $C$ 是比深度和宽度更有效的维度。\n\n### ImageNet-1K 模型表现\n\n![img5](/content/papers/assets/519d441ad166763fc8b8b33bd4d5850063e7ab1ed919663c3d53af746f07bd6e.jpg)\n\n上表显示了在 ImageNet 验证集上进行单次裁切测试的结果。\n\nResNeXt 的结果与 ResNet、Inception-v3/v4 和 Inception-ResNet-v2 相比毫不逊色；ResNeXt-101 (64×4d) 以 320×320 单次裁切达到 4.4% 的 top-5 错误率。\n\n不仅如此，ResNeXt 的架构设计比所有 Inception 模型简单得多，手动设置的超参数也少得多。\n\n## 结论\n\n本论文提出了一种简单且高度模块化的网络架构 ResNeXt，通过重复聚合一组具有相同拓扑结构的变换构建而成，形成了一种同质的多分支架构，只需设置少量的超参数。\n\n作者引入了一个新的维度：基数（cardinality），作为除了深度和宽度之外的重要因素。\n\n实验结果表明，在保持模型复杂度的条件下，增加基数能够显著提高分类准确率。\n\n相比于单纯增加模型的深度或宽度，增加基数更为有效。\n\n\n> [!TIP]\n> \n> 这份论文为之后的可分离卷积、MobileNet 等轻量化模型的发展提供了重要的参考。\n","imageSizes":{"/content/papers/assets/268f5b641a1a85d929d88f2aa0d636902dd9d101824284c32266668493563672.jpg":{"width":1224,"height":576},"/content/papers/assets/28910e4d22dc8f2c055dc119b8fbd9728731c45a041ef67bacc24c4ad7e6301c.jpg":{"width":1632,"height":506},"/content/papers/assets/e30a91af00c174fae7a49cc498f42a513476128d1f427ba921c50eb7d595d9c2.jpg":{"width":1044,"height":1024},"/content/papers/assets/f1b9572741682865800ae1649291b9939705a8639603bcaa4d99e3cb547a3feb.jpg":{"width":1356,"height":580},"/content/papers/assets/8c576fcc821cb0a29fa73df0db77e811a3ff980ee24848c132c2f5ab38550dc5.jpg":{"width":1110,"height":594},"/content/papers/assets/519d441ad166763fc8b8b33bd4d5850063e7ab1ed919663c3d53af746f07bd6e.jpg":{"width":1208,"height":544}}}
