{"notes":[{"id":"lightweight/mobilenet-v3","category":"lightweight","legacyPath":"lightweight/1905-mobilenet-v3","legacySlug":"1905-mobilenet-v3","legacyPaths":[],"title":"MobileNet-V3","subtitle":"客製化搜索架構","author":"Z. Yuan","paperDate":"2019-05-06","publishedOn":"2024-06-29","paperTitle":"Searching for MobileNetV3","paperUrl":"https://arxiv.org/abs/1905.02244","bibliography":{"title":"Searching for MobileNetV3","authors":["Andrew Howard","Mark Sandler","Grace Chu","Liang-Chieh Chen","Bo Chen","Mingxing Tan","Weijun Wang","Yukun Zhu","Ruoming Pang","Vijay Vasudevan","Quoc V. Le","Hartwig Adam"],"url":"https://arxiv.org/abs/1905.02244","pdfUrl":"https://arxiv.org/pdf/1905.02244","codeUrl":null,"preprint":{"id":"1905.02244","submittedOn":"2019-05-06","url":"https://arxiv.org/abs/1905.02244"},"publication":{"venue":"ICCV","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1905.02244"},"citation":"@misc{lightweightmobilenetv32019,\n  title = {{Searching for MobileNetV3}},\n  author = {Andrew Howard and Mark Sandler and Grace Chu and Liang-Chieh Chen and Bo Chen and Mingxing Tan and Weijun Wang and Yukun Zhu and Ruoming Pang and Vijay Vasudevan and Quoc V. Le and Hartwig Adam},\n  year = {2019},\n  eprint = {1905.02244},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1905.02244}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v3.78d376f40aa4.bib"},"thumbnail":{"src":"/content/papers/assets/09d4bf8776407cb2ac13bc206505f30314b21d2459add8129ffcb929d11ec1bf.jpg","width":1552,"height":588},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/mobilenet-v3.ca9bddbc449b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"mobilenet-v2-模組","text":"MobileNet-V2 模組","depth":3},{"id":"mobilenet-v3-模組","text":"MobileNet-V3 模組","depth":3},{"id":"網路架構搜索","text":"網路架構搜索","depth":3},{"id":"網路結尾變更","text":"網路結尾變更","depth":3},{"id":"縮減初始濾波器","text":"縮減初始濾波器","depth":3},{"id":"非線性啟動函數","text":"非線性啟動函數","depth":3},{"id":"網路架構大","text":"網路架構（大）","depth":3},{"id":"網路架構小","text":"網路架構（小）","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-實驗結果","text":"ImageNet 實驗結果","depth":3},{"id":"移動裝置評估","text":"移動裝置評估","depth":3},{"id":"優化紀錄","text":"優化紀錄","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v2","relation":"background","topic":"行動裝置上的高效率卷積"},{"id":"lightweight/mobilenet-v4","relation":"next","topic":"行動裝置上的高效率卷積"},{"id":"lightweight/senet","relation":"comparison","topic":"輕量網路裡的通道注意力"}]},{"id":"lightweight/mobilenet-v2","category":"lightweight","legacyPath":"lightweight/1801-mobilenet-v2","legacySlug":"1801-mobilenet-v2","legacyPaths":[],"title":"MobileNet-V2","subtitle":"瓶頸內的淬煉","author":"Z. Yuan","paperDate":"2018-01-13","publishedOn":"2024-06-27","paperTitle":"MobileNetV2: Inverted Residuals and Linear Bottlenecks","paperUrl":"https://arxiv.org/abs/1801.04381","bibliography":{"title":"MobileNetV2: Inverted Residuals and Linear Bottlenecks","authors":["Mark Sandler","Andrew Howard","Menglong Zhu","Andrey Zhmoginov","Liang-Chieh Chen"],"url":"https://arxiv.org/abs/1801.04381","pdfUrl":"https://arxiv.org/pdf/1801.04381","codeUrl":null,"preprint":{"id":"1801.04381","submittedOn":"2018-01-13","url":"https://arxiv.org/abs/1801.04381"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1801.04381"},"citation":"@misc{lightweightmobilenetv22018,\n  title = {{MobileNetV2: Inverted Residuals and Linear Bottlenecks}},\n  author = {Mark Sandler and Andrew Howard and Menglong Zhu and Andrey Zhmoginov and Liang-Chieh Chen},\n  year = {2018},\n  eprint = {1801.04381},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.04381}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v2.a882707cbaa2.bib"},"thumbnail":{"src":"/content/papers/assets/e6bbfb346bb449f83897d76b69c3711d2e9937d40320a7aab0f3843653a688c0.jpg","width":2332,"height":1324},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/mobilenet-v2.a7a89cbedac8.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"線性瓶頸","text":"線性瓶頸","depth":3},{"id":"殘差網路","text":"殘差網路","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"反轉殘差","text":"反轉殘差","depth":3},{"id":"擴張瓶頸論文附錄-a","text":"擴張瓶頸（論文附錄 A）","depth":3},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"圖像分類效能","text":"圖像分類效能","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v1","relation":"background","topic":"行動裝置上的高效率卷積"},{"id":"lightweight/mobilenet-v3","relation":"next","topic":"行動裝置上的高效率卷積"}]},{"id":"lightweight/mobilenet-v4","category":"lightweight","legacyPath":"lightweight/2404-mobilenet-v4","legacySlug":"2404-mobilenet-v4","legacyPaths":[],"title":"MobileNet-V4","subtitle":"時隔五年的傳承","author":"Z. Yuan","paperDate":"2024-04-16","publishedOn":"2024-06-29","paperTitle":"MobileNetV4 -- Universal Models for the Mobile Ecosystem","paperUrl":"https://arxiv.org/abs/2404.10518","bibliography":{"title":"MobileNetV4 -- Universal Models for the Mobile Ecosystem","authors":["Danfeng Qin","Chas Leichner","Manolis Delakis","Marco Fornoni","Shixin Luo","Fan Yang","Weijun Wang","Colby Banbury","Chengxi Ye","Berkin Akin","Vaibhav Aggarwal","Tenghui Zhu","Daniele Moro","Andrew Howard"],"url":"https://arxiv.org/abs/2404.10518","pdfUrl":"https://arxiv.org/pdf/2404.10518","codeUrl":null,"preprint":{"id":"2404.10518","submittedOn":"2024-04-16","url":"https://arxiv.org/abs/2404.10518"},"publication":{"venue":"ECCV","year":2024,"kind":"conference","url":"https://doi.org/10.1007/978-3-031-73661-2_5"},"citation":"@misc{lightweightmobilenetv42024,\n  title = {{MobileNetV4 -- Universal Models for the Mobile Ecosystem}},\n  author = {Danfeng Qin and Chas Leichner and Manolis Delakis and Marco Fornoni and Shixin Luo and Fan Yang and Weijun Wang and Colby Banbury and Chengxi Ye and Berkin Akin and Vaibhav Aggarwal and Tenghui Zhu and Daniele Moro and Andrew Howard},\n  year = {2024},\n  eprint = {2404.10518},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2404.10518}\n}\n","citationPath":"/content/papers/citations/lightweight/mobilenet-v4.e846766e9e65.bib"},"thumbnail":{"src":"/content/papers/assets/0a44fc5e9bc8cbe6aa067e68dcc4a12271a4a731e2b7af226ea0e0308ea2fd66.jpg","width":1324,"height":504},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/mobilenet-v4.72cf9bcf3edb.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"混合注意力","text":"混合注意力","depth":3},{"id":"高效注意力","text":"高效注意力","depth":3},{"id":"不同的卷積網路架構","text":"不同的卷積網路架構","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"網路架構","text":"網路架構","depth":3},{"id":"mobile-mqa","text":"Mobile MQA","depth":3},{"id":"跨裝置最優化","text":"跨裝置最優化","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-的實驗結果","text":"ImageNet 的實驗結果","depth":3},{"id":"mqa-的效果","text":"MQA 的效果","depth":3},{"id":"mnv4-conv-s","text":"MNv4-Conv-S","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v3","relation":"background","topic":"行動裝置上的高效率卷積"}]},{"id":"lightweight/senet","category":"lightweight","legacyPath":"lightweight/1709-senet","legacySlug":"1709-senet","legacyPaths":[],"title":"SENet","subtitle":"擠！用力擠！","author":"Z. Yuan","paperDate":"2017-09-05","publishedOn":"2024-06-28","paperTitle":"Squeeze-and-Excitation Networks","paperUrl":"https://arxiv.org/abs/1709.01507","bibliography":{"title":"Squeeze-and-Excitation Networks","authors":["Jie Hu","Li Shen","Samuel Albanie","Gang Sun","Enhua Wu"],"url":"https://arxiv.org/abs/1709.01507","pdfUrl":"https://arxiv.org/pdf/1709.01507","codeUrl":null,"preprint":{"id":"1709.01507","submittedOn":"2017-09-05","url":"https://arxiv.org/abs/1709.01507"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1709.01507"},"citation":"@misc{lightweightsenet2017,\n  title = {{Squeeze-and-Excitation Networks}},\n  author = {Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},\n  year = {2017},\n  eprint = {1709.01507},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1709.01507}\n}\n","citationPath":"/content/papers/citations/lightweight/senet.1766a7978dde.bib"},"thumbnail":{"src":"/content/papers/assets/322f16ed8c782ffbdd7d7db3e8292c991908e3543dc69e1addd9238e22484363.jpg","width":1688,"height":394},"locale":"zh-hant","path":"/content/papers/zh-hant/lightweight/senet.dd5fabe0868c.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"影像分類效果展示","text":"影像分類效果展示","depth":3},{"id":"輕量模型效果展示","text":"輕量模型效果展示","depth":3},{"id":"squeeze-的重要性","text":"Squeeze 的重要性","depth":3},{"id":"excitation-的重要性","text":"Excitation 的重要性","depth":3},{"id":"其他實驗結果","text":"其他實驗結果","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"lightweight/mobilenet-v3","relation":"comparison","topic":"輕量網路裡的通道注意力"},{"id":"lightweight/pp-lcnet","relation":"comparison","topic":"輕量網路裡的通道注意力"}]}],"text":"前些日子，SENet 幾乎就要提出 MobileNet-V3 了！\n\n可惜在最後時刻點到即止，只留下了論文的一個小段落。\n\n- [**SENet-輕量模型效果展示**](/blog/papers/lightweight/senet#輕量模型效果展示)\n\nGoogle 團隊顯然也想繼續推進 MobileNet 的發展，於是他們開始了一場關於 MobileNet-V3 的探索之旅。\n\n## 定義問題\n\n其實也沒什麼問題。\n\n就是作者們想要找到一個更好的 MobileNet。\n\n於是他們想了兩個方向：\n\n1. **使用網路架構搜尋技術**：也就是 NAS（神經網路架構搜索）。\n2. **使用 NetAdapt 演算法**：用來針對手機 CPU 進行調整。\n\n## 解決問題\n\n### MobileNet-V2 模組\n\n\n\n![MobileNet-V2 模組](/content/papers/assets/b4d50b363f465156b76045253719a991f91fac46f2e3a25722746da6d8862f2a.jpg)\n\n\n\n### MobileNet-V3 模組\n\n\n\n![MobileNet-V3 模組](/content/papers/assets/09d4bf8776407cb2ac13bc206505f30314b21d2459add8129ffcb929d11ec1bf.jpg)\n\n\n\n對比上下兩張架構圖，你能看出來 MobileNet-V3 的改進嗎？\n\n其中一個差別是：在擴展後的深度卷積之後，加入 `Squeeze-and-Excitation` 模組，對較寬的通道表示加權。\n\n在約一年前提出的論文 `MnasNet` 中，已經在 `MobileNet-V2` 的基礎上加入了 `Squeeze-and-Excitation` 模組。\n\n- [**[18.07] MnasNet: Platform-Aware Neural Architecture Search for Mobile**](https://arxiv.org/abs/1807.11626)\n\n在 `MobileNet-V3` 中，這個模組進一步將較難在手機上高效計算的 `Sigmoid` 換成 `Hard-Sigmoid`；部分卷積層也改用 `Hard-Swish`，但並非每一層都使用這兩種函數。\n\n\n> [!TIP]\n> \n> 很多人都以為 `Squeeze-and-Excitation` 模組是由 `MobileNet-V3` 提出的，不對唷！\n\n\n### 網路架構搜索\n\n新的模組已經準備好了，接下來的問題就是：怎麼疊呢？\n\n這一個部分沿用了 `MnasNet` 的工作方法，使用基於 RNN 的控制器來搜索網路架構。較大的模型直接以 MnasNet-A1 為起點；較小的模型則調整延遲獎勵權重（$-0.07$ 改為 $-0.15$），重新搜索起始架構。\n\n接著用 `NetAdapt` 逐層調整通道數，再進行其他架構最佳化。下圖取自 NetAdapt 原論文，圖中的延遲與能耗數字不是 MobileNet-V3 的實驗結果。\n\n- [**[18.04] Netadapt: Platform-aware neural network adaptation for mobile applications**](https://arxiv.org/abs/1804.03230)\n\n\n\n   ![NetAdapt 演算法](/content/papers/assets/d7fe74f3b6558378da85444ddc4acb464a039b3db7b8f551f8cf69cea197fa4d.jpg)\n\n\n\n\n> [!TIP]\n> \n> 這個部分不是本文的重點，我們只要知道這個部分是用來調整網路架構的就好。\n\n\n### 網路結尾變更\n\n![MobileNet-V3 結尾設計](/content/papers/assets/2a773272e73eb497269a3b1fef822a71b44858be5458ca14b1d2b7fe4044b65a.jpg)\n\n基於 MobileNetV2 的反向瓶頸結構和變體的模型使用 1x1 卷積作為最後一層，以便擴展到更高維度的特徵空間。也就是上圖中 `Original Last Stage` 內的 1280 維度的部分。\n\n為了擁有豐富的預測特徵，這一層很重要，但這是以額外延遲為代價的。\n\n為了減少延遲並保留特徵，作者將這一層移動到全局平均池化「之後」，也就是上圖中 `Efficient Last Stage` 的樣子。\n\n原本計算時採用 7x7 的解析度，經過調整後，就是在 1x1 的解析度上進行計算。作者也因此移除了前一個瓶頸單元中不再需要的投影與深度卷積；這組尾端修改在論文中減少約 3000 萬次乘加運算與 7 ms 延遲，準確率幾乎不變。\n\n### 縮減初始濾波器\n\n作者觀察到初始濾波器常有互為「鏡像」的情形，因此嘗試縮減這層的通道數。\n\n作者將初始濾波器從 32 減少到 16，並選用 `hard-swish` 啟動函數；實驗中準確率維持不變，延遲減少約 2 ms，乘加運算減少約 1000 萬次。\n\n### 非線性啟動函數\n\n![activate](/content/papers/assets/19fe666ec5e382660978b139478cdd1806e603c0e4018947a62794c1c26d3d98.jpg)\n\n作者在論文中大量採用了 `hard-swish` 和 `hard-sigmoid` 這兩種啟動函數。\n\n這裡簡單看一下這些啟動函數的差異：\n\n1. **ReLU (Rectified Linear Unit)**\n\n   - **定義**：$ f(x) = \\max(0, x) $\n   - **特點**：當輸入 $ x $ 為正數時，輸出就是 $ x $；當輸入 $ x $ 為負數時，輸出為 0。\n   - **用途**：由於其計算簡單和非飽和性質，ReLU 在許多深度學習模型中被廣泛使用。\n\n2. **Swish**\n\n   - **定義**：$ f(x) = x \\cdot \\sigma(x) $，其中 $ \\sigma(x) $ 是 Sigmoid 函數 $ \\sigma(x) = \\frac{1}{1 + e^{-x}} $。\n   - **特點**：Swish 函數是自我閘化的啟動函數，它結合了線性和非線性特性。\n   - **用途**：在某些深度學習任務中，相較於 ReLU 和其他啟動函數，Swish 顯示了更好的性能。\n\n3. **Hard-Swish**\n\n   - **定義**：$ f(x) = x \\cdot \\text{relu6}(x+3)/6 $，其中 $\\text{relu6}(x) = \\min(\\max(0,x), 6)$。\n   - **特點**：Hard-Swish 是 Swish 函數的一種分段近似；論文使用最佳化實作來降低手機上的計算延遲。\n   - **用途**：常用於需要高效計算的應用場景，如移動設備或邊緣裝置上的神經網路模型。\n\n4. **Sigmoid**\n\n   - **定義**：$ f(x) = \\frac{1}{1 + e^{-x}} $\n   - **特點**：輸出範圍在 0 到 1 之間，常用於將輸出壓縮到一個限定範圍內。\n   - **用途**：由於其良好的概率解釋，經常用於二分類問題的最後輸出層。\n\n5. **Hard-Sigmoid**\n   - **定義**：$ f(x) = \\text{relu6}(x+3)/6 $\n   - **特點**：這是 Sigmoid 函數的一種簡化版，其計算速度更快，適合於執行效率要求高的場合。\n   - **用途**：與 Sigmoid 類似，但更常用於需要高效計算的模型中。\n\n### 網路架構（大）\n\n經過了上面的規畫，透過網路架構搜索、逐層調整與手動修改，得到 MobileNetV3-Large。\n\n下表的讀法，大部分和 `MobileNet-V2` 一樣，新的地方是：\n\n- **SE**：表示是否使用了 `Squeeze-and-Excitation` 模組。\n- **NL**：表示使用哪一種非線性啟動函數。\n- **HS**：表示使用 `hard-swish` 啟動函數。\n- **RE**：表示使用 `ReLU` 啟動函數。\n- **NBN**：表示沒有使用 `Batch Normalization`。\n\n\n\n![MobileNet-V3 large](/content/papers/assets/ba2a0a168022083c8a5e214ff679a376f044dd7e0862080c21a334be17cc276f.jpg)\n\n\n\n### 網路架構（小）\n\n針對小型模型，作者們重新搜索起始架構，再以相同流程調整出 MobileNetV3-Small。\n\n\n\n![MobileNet-V3 small](/content/papers/assets/77cea0496314764ae8d4910b7c51503b792bd65b97fc932e30690517f28c0a1a.jpg)\n\n\n\n## 討論\n\n### ImageNet 實驗結果\n\n\n\n![ImageNet 實驗結果](/content/papers/assets/74dd95768d9a993b1ba0a5d9be88c986b479e700718e744eec31559366701887.jpg)\n\n\n\n圖中橫軸是 Pixel 1 單核心延遲，縱軸是 ImageNet top-1 準確率。在圖示的模型設定與延遲範圍內，MobileNetV3-Large 與 Small 都提供了比圖中的 MnasNet、ProxylessNAS 和 MobileNetV2 對照點更好的取捨。\n\n### 移動裝置評估\n\n\n\n![MobileNet-V3 實驗結果](/content/papers/assets/51319b69bcafe958665c6a556b2e53026f0544942b8507b5f40bd20e88f87297.jpg)\n\n\n\n作者在上表中報告了不同 Pixel 手機上的浮點效能。\n\n上表中 P-1 表示 Pixel 1，P-2 表示 Pixel 2，P-3 表示 Pixel 3；三欄是浮點模型以單一大核心、batch size 1 測得的推論延遲，單位為 ms，數值越低越快。例如 V3-Large 1.0 在 Pixel 1 為 51 ms、top-1 準確率 75.2%，相較 V2 1.0 的 64 ms、72.0%。\n\n### 優化紀錄\n\n\n\n![MobileNet-V3 優化紀錄](/content/papers/assets/7c68decdd64a2bab9d063ef325cb903c1a8f4126b0cf044e6aba74a542036c31.jpg)\n\n\n\n上圖展示了作者們在本論文中的準確率與延遲取捨。\n\n從 MnasNet 出發，加入 `SE` 與 `h-swish` 的點提高了準確率；`NetAdapt` 與精簡尾端則主要把延遲往左推。圖中的點呈現各項修改的影響，並非每一步都同時提升準確率與速度。\n\n## 結論\n\n移動設備的市場很大，算力有限，因此需要更「客製化」的模型。\n\n這篇論文展示了如何通過網路架構搜索和 NetAdapt 演算法，在 Pixel 手機上實現了更好的效能。\n\n未來或許能夠拓展到 Pixel、Apple、Samsung 各種不同手機上的統一架構。\n\n不管是對手機使用者，還是對我們這種模型開發者，這都是一個好消息。\n","imageSizes":{"/content/papers/assets/b4d50b363f465156b76045253719a991f91fac46f2e3a25722746da6d8862f2a.jpg":{"width":1224,"height":552},"/content/papers/assets/09d4bf8776407cb2ac13bc206505f30314b21d2459add8129ffcb929d11ec1bf.jpg":{"width":1552,"height":588},"/content/papers/assets/d7fe74f3b6558378da85444ddc4acb464a039b3db7b8f551f8cf69cea197fa4d.jpg":{"width":1224,"height":624},"/content/papers/assets/2a773272e73eb497269a3b1fef822a71b44858be5458ca14b1d2b7fe4044b65a.jpg":{"width":1226,"height":530},"/content/papers/assets/19fe666ec5e382660978b139478cdd1806e603c0e4018947a62794c1c26d3d98.jpg":{"width":1422,"height":390},"/content/papers/assets/ba2a0a168022083c8a5e214ff679a376f044dd7e0862080c21a334be17cc276f.jpg":{"width":1148,"height":1052},"/content/papers/assets/77cea0496314764ae8d4910b7c51503b792bd65b97fc932e30690517f28c0a1a.jpg":{"width":1124,"height":868},"/content/papers/assets/74dd95768d9a993b1ba0a5d9be88c986b479e700718e744eec31559366701887.jpg":{"width":1224,"height":490},"/content/papers/assets/51319b69bcafe958665c6a556b2e53026f0544942b8507b5f40bd20e88f87297.jpg":{"width":1224,"height":544},"/content/papers/assets/7c68decdd64a2bab9d063ef325cb903c1a8f4126b0cf044e6aba74a542036c31.jpg":{"width":1224,"height":604}}}
