{"notes":[{"id":"lightweight/mobilenet-v1","category":"lightweight","legacyPath":"lightweight/1704-mobilenet-v1","legacySlug":"1704-mobilenet-v1","legacyPaths":[],"title":"MobileNet-V1","subtitle":"深度可分离卷积的发扬者","author":"Z. Yuan","paperDate":"2017-04-17","publishedOn":"2024-06-25","paperTitle":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","paperUrl":"https://arxiv.org/abs/1704.04861","bibliography":{"title":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications","authors":["Andrew G. Howard","Menglong Zhu","Bo Chen","Dmitry Kalenichenko","Weijun Wang","Tobias Weyand","Marco Andreetto","Hartwig Adam"],"url":"https://arxiv.org/abs/1704.04861","pdfUrl":"https://arxiv.org/pdf/1704.04861","codeUrl":null,"preprint":{"id":"1704.04861","submittedOn":"2017-04-17","url":"https://arxiv.org/abs/1704.04861"},"publication":null,"citation":"@misc{lightweightmobilenetv12017,\n  title = {{MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications}},\n  author = {Andrew G. Howard and Menglong Zhu and Bo Chen and Dmitry Kalenichenko and Weijun Wang and Tobias Weyand and Marco Andreetto and Hartwig Adam},\n  year = {2017},\n  eprint = {1704.04861},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1704.04861}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v1.e9c4c525fed4.bib"},"thumbnail":{"src":"/content/papers/assets/b8390bd8e0f21f1b1eaa6b2a5d4a1b0d9eba36e2fe5dd0878830d757839d30f1.jpg","width":1064,"height":1080},"locale":"zh-hans","path":"/content/papers/zh-hans/lightweight/mobilenet-v1.aef841466ce6.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"深度可分离卷积","text":"深度可分离卷积","depth":3},{"id":"网络架构","text":"网络架构","depth":2},{"id":"训练技巧","text":"训练技巧","depth":2},{"id":"讨论","text":"讨论","depth":2},{"id":"深度可分离卷积的性能","text":"深度可分离卷积的性能","depth":3},{"id":"要深度还是要宽度","text":"要深度还是要宽度？","depth":3},{"id":"和其他模型比较","text":"和其他模型比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v2","relation":"next","topic":"移动设备上的高效卷积"}]},{"id":"lightweight/mobilenet-v2","category":"lightweight","legacyPath":"lightweight/1801-mobilenet-v2","legacySlug":"1801-mobilenet-v2","legacyPaths":[],"title":"MobileNet-V2","subtitle":"瓶颈内的淬炼","author":"Z. Yuan","paperDate":"2018-01-13","publishedOn":"2024-06-27","paperTitle":"MobileNetV2: Inverted Residuals and Linear Bottlenecks","paperUrl":"https://arxiv.org/abs/1801.04381","bibliography":{"title":"MobileNetV2: Inverted Residuals and Linear Bottlenecks","authors":["Mark Sandler","Andrew Howard","Menglong Zhu","Andrey Zhmoginov","Liang-Chieh Chen"],"url":"https://arxiv.org/abs/1801.04381","pdfUrl":"https://arxiv.org/pdf/1801.04381","codeUrl":null,"preprint":{"id":"1801.04381","submittedOn":"2018-01-13","url":"https://arxiv.org/abs/1801.04381"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1801.04381"},"citation":"@misc{lightweightmobilenetv22018,\n  title = {{MobileNetV2: Inverted Residuals and Linear Bottlenecks}},\n  author = {Mark Sandler and Andrew Howard and Menglong Zhu and Andrey Zhmoginov and Liang-Chieh Chen},\n  year = {2018},\n  eprint = {1801.04381},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.04381}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v2.a882707cbaa2.bib"},"thumbnail":{"src":"/content/papers/assets/e6bbfb346bb449f83897d76b69c3711d2e9937d40320a7aab0f3843653a688c0.jpg","width":2332,"height":1324},"locale":"zh-hans","path":"/content/papers/zh-hans/lightweight/mobilenet-v2.3cc14e69626b.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"线性瓶颈","text":"线性瓶颈","depth":3},{"id":"残差网络","text":"残差网络","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"反转残差","text":"反转残差","depth":3},{"id":"扩张瓶颈论文附录-a","text":"扩张瓶颈（论文附录 A）","depth":3},{"id":"模型架构","text":"模型架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"图像分类性能","text":"图像分类性能","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v1","relation":"background","topic":"移动设备上的高效卷积"},{"id":"lightweight/mobilenet-v3","relation":"next","topic":"移动设备上的高效卷积"}]}],"text":"深度可分离卷积是一种卷积神经网络的结构，它将标准卷积分解为两个独立的层：深度卷积和逐点卷积。\n\n这种结构可以减少模型的参数数量和计算量，从而使得模型更加轻量化，这对于移动设备等资源受限的场景非常有用。\n\n虽然这篇论文通篇在讲深度可分离卷积，但实际上这个结构的创始者却是另有其人：\n\n- [**Rigid-motion scattering for image classification**](https://arxiv.org/abs/1403.1687)\n\nMobileNets 的作者也提到，他们的主要贡献在于将「深度可分离卷积」这个结构，系统性地应用到了卷积神经网络中，并且在 ImageNet 上取得了不错的效果。\n\n也就是现在赫赫有名的： **MobileNet-V1**。\n\n## 定义问题\n\n在这几年的发展中，人们对于构建小型且高效的神经网络愈来愈感兴趣。\n\n但是，许多关于小型网络的论文「只关注大小而不考虑效率」。\n\n显然地，又小又快的网络架构，才是我们真正需要的。\n\n\n> [!TIP]\n> \n> 虽然在论文中没有明确提到，但我们猜测这句话指的是 [**SqueezeNet**](https://arxiv.org/abs/1602.07360)。\n\n\n## 解决问题\n\n### 深度可分离卷积\n\n\n\n![Depthwise Separable Convolution](/content/papers/assets/b8390bd8e0f21f1b1eaa6b2a5d4a1b0d9eba36e2fe5dd0878830d757839d30f1.jpg)\n\n\n\n传统卷积层将输入的特征图通过一个多通道的卷积核过滤并结合，生成新的特征图。这个过程在每个输出通道上都会考虑所有输入通道的信息，计算成本很高。而深度可分离卷积则将此过程分解为两个阶段，先进行深度卷积，即每个输入通道独立应用一个滤波器，只处理输入数据的一个通道，然后通过逐点卷积将这些输出通道的结果进行组合。\n\n深度卷积不会改变输入特征图的通道数，只对每个通道进行空间上的过滤操作。每个通道使用一个独立的卷积核，这意味着如果输入特征图有 M 个通道，那么就有 M 个深度卷积核。逐点卷积则是使用 1×1 的卷积核来对这些过滤后的通道进行线性组合，产生最终的输出通道。这种分解方式的主要优势是显著降低了计算量。\n\n对于深度卷积，其计算成本为$DK \\times DK \\times M \\times DF \\times DF$，其中$DK$ 是卷积核的空间尺寸，$DF$ 是特征图的空间尺寸，$M$ 是输入通道数；此处沿用论文输入与输出尺寸相同的假设。\n\n对于逐点卷积，计算成本则为$M \\times N \\times DF \\times DF$，其中$N$ 是输出通道数。\n\n将这两个步骤的成本相加，仍然远低于传统卷积的$DK \\times DK \\times M \\times N \\times DF \\times DF$。\n\n因此，使用深度可分离卷积可以在保持相对较高准确度的同时，显著减少模型的计算需求和参数量，非常适合需要在计算资源有限的设备上运行的应用。\n\n\n> [!NOTE]\n> \n> \n> 我们可以顺手用 PyTorch 实现一下深度可分离卷积：\n> \n> ```python\n> import torch\n> import torch.nn as nn\n> import torch.nn.functional as F\n> \n> class DepthwiseSeparableConv(nn.Module):\n> \n>     def __init__(self, n_in, n_out, kernel_size, padding=0, stride=1):\n>         super(DepthwiseSeparableConv, self).__init__()\n>         self.depthwise = nn.Conv2d(n_in, n_in, kernel_size=kernel_size, padding=padding, stride=stride, groups=n_in)\n>         self.pointwise = nn.Conv2d(n_in, n_out, kernel_size=1)\n> \n>     def forward(self, x):\n>         out = self.depthwise(x)\n>         out = self.pointwise(out)\n>         return out\n> ```\n> \n> 在这篇论文的实现中，有提到每个卷积层后面都接了 Batch Normalization 和 ReLU 激活函数，如下图：\n> \n> \n> \n> ![Depthwise Separable Convolution with BN and ReLU](/content/papers/assets/a272daf6d5a3edb79671be1e681681eae0378c5bbf06620446ec957194e73c46.jpg)\n> \n> \n> \n> 所以实现方式可以改成这样：\n> \n> ```python\n> import torch\n> import torch.nn as nn\n> import torch.nn.functional as F\n> \n> class DepthwiseSeparableConv(nn.Module):\n> \n>     def __init__(self, n_in, n_out, kernel_size, padding=0, stride=1):\n>         super(DepthwiseSeparableConv, self).__init__()\n>         self.depthwise = nn.Conv2d(n_in, n_in, kernel_size=kernel_size, padding=padding, stride=stride, groups=n_in)\n>         self.pointwise = nn.Conv2d(n_in, n_out, kernel_size=1)\n>         self.depth_bn = nn.BatchNorm2d(n_in)\n>         self.depth_relu = nn.ReLU()\n>         self.point_bn = nn.BatchNorm2d(n_out)\n>         self.point_relu = nn.ReLU()\n> \n>     def forward(self, x):\n>         x = self.depthwise(x)\n>         x = self.depth_bn(x)\n>         x = self.depth_relu(x)\n>         x = self.pointwise(x)\n>         x = self.point_bn(x)\n>         x = self.point_relu(x)\n>         return x\n> ```\n> \n\n\n## 网络架构\n\n根据论文，MobileNet-V1 的网络架构，「**只有第一层的 3x3 是一般卷积核**」之外，其他都是由深度可分离卷积堆叠而成。\n\n> The MobileNet structure is built on depthwise separable convolutions as mentioned in the previous section except for the first layer which is a full convolution.\n\n整体结构如下表：\n\n\n\n![MobileNet-V1 Architecture](/content/papers/assets/96bb0fb7d329b112dde22c903e2fa32642a4a29b8e007b480d6ef944d054f332.jpg)\n\n\n\n## 训练技巧\n\nMobileNet-V1 的作者在训练过程中使用了一些技巧来提高模型的性能：\n\n- 使用较少的正则化和数据增强，因为小模型的过拟合问题较少。\n- 在深度可分离卷积的滤波器上，放置很少或不放置任何权重衰减（L2 正则化）。\n- 使用 RMSprop 作为优化器。\n- 采用类似 Inception-V3 的异步梯度下降。\n\n\n> [!TIP]\n> \n> 异步梯度下降（Asynchronous Gradient Descent）是一种用于深度学习训练中的优化技术，特别适用于分布式计算环境。在传统的同步梯度下降中，所有计算节点（或工作器）必须完成它们的梯度计算后，才能进行模型更新。这意味着整个训练过程的速度受到最慢节点的限制。相比之下，异步梯度下降允许每个计算节点独立地计算梯度并更新共享模型，而不需要等待其他节点的计算结果。\n\n\n## 讨论\n\n在整体的实验设计中，上述提到的网络架构做为 Baseline，作者另外引入两个常数的超参数：\n\n- **Width Multiplier $\\alpha$**：用来控制网络的宽度，即各层的输入与输出通道数。\n- **Resolution Multiplier $\\rho$**：用来控制输入图像的分辨率，进而缩小各层特征图的空间尺寸。\n\n### 深度可分离卷积的性能\n\n\n\n![Depthwise Separable Convolution Performance](/content/papers/assets/142ea49129c865e2e093bb994cd15ad68055f430d35b314d6ce70fa95923a37c.jpg)\n\n\n\n与全卷积版本相比，使用深度可分离卷积的 ImageNet 准确率从 71.7% 降至 70.6%，差 1.1 个百分点，乘加运算与参数量也大幅减少。\n\n### 要深度还是要宽度？\n\n\n\n![Depth vs Width](/content/papers/assets/ea604ea51943d65149e5f66724ca777716c27b9d876a7b5256f3ff2fafde6823.jpg)\n\n\n\n表中计算量相近的两个模型，0.75 倍宽度版本的准确率为 68.4%，删去五层的浅层版本为 65.3%，相差 3.1 个百分点。\n\n在这组比较中，算力受限时缩窄模型比删去这五层保留了更高的准确率。\n\n### 和其他模型比较\n\n\n\n![Comparison with Other Models](/content/papers/assets/751b301a1b2d2396b0200cc00a8cf8d1dd58523e06a386f3a4f0cb219acf4d07.jpg)\n\n\n\n将完整的 MobileNet 与原始的 GoogLeNet 和 VGG16 进行了比较。\n\nMobileNet 几乎与 VGG16 一样准确，同时体积小 32 倍，计算量降低 27 倍。而且比 GoogLeNet 更准确，同时体积更小，计算量减少 2.5 倍以上。\n\n在大小相近的情况下，0.50 MobileNet-160 的乘加运算量约为 SqueezeNet 的 1/22，ImageNet 准确率高 2.7 个百分点。\n\n## 结论\n\nMobileNet-V1 的推出，将深度可分离卷积技术正式地带入了大众的视野。这种结构在保持相对高准确度的同时，大幅降低了模型的大小和计算需求，从而解决了在资源受限的环境上运行复杂深度学习模型的挑战。\n\n这篇论文不仅提供了一种在资源有限环境中部署深度学习模型的有效方案，而且其设计理念启发了后续多种轻量级深度学习架构的开发，进一步推动了深度学习架构在移动设备和边缘计算中的应用。\n\n\n> [!TIP]\n> \n> MobileNet 在后续也推出 V2、V3 和 V4 版本，进一步改进了模型的性能和效率。这一系列的研究成果在实际应用中取得了广泛的成功，成为了轻量级深度学习模型的经典之作。\n","imageSizes":{"/content/papers/assets/b8390bd8e0f21f1b1eaa6b2a5d4a1b0d9eba36e2fe5dd0878830d757839d30f1.jpg":{"width":1064,"height":1080},"/content/papers/assets/a272daf6d5a3edb79671be1e681681eae0378c5bbf06620446ec957194e73c46.jpg":{"width":972,"height":504},"/content/papers/assets/96bb0fb7d329b112dde22c903e2fa32642a4a29b8e007b480d6ef944d054f332.jpg":{"width":932,"height":1080},"/content/papers/assets/142ea49129c865e2e093bb994cd15ad68055f430d35b314d6ce70fa95923a37c.jpg":{"width":988,"height":292},"/content/papers/assets/ea604ea51943d65149e5f66724ca777716c27b9d876a7b5256f3ff2fafde6823.jpg":{"width":972,"height":284},"/content/papers/assets/751b301a1b2d2396b0200cc00a8cf8d1dd58523e06a386f3a4f0cb219acf4d07.jpg":{"width":1012,"height":684}}}
