MobileNetV4 -- Universal Models for the Mobile Ecosystem
- Publication
- ECCV 2024Conference
- First submitted to arXiv
- arXiv:2404.10518
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{lightweightmobilenetv42024,
title = {{MobileNetV4 -- Universal Models for the Mobile Ecosystem}},
author = {Danfeng Qin and Chas Leichner and Manolis Delakis and Marco Fornoni and Shixin Luo and Fan Yang and Weijun Wang and Colby Banbury and Chengxi Ye and Berkin Akin and Vaibhav Aggarwal and Tenghui Zhu and Daniele Moro and Andrew Howard},
year = {2024},
eprint = {2404.10518},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2404.10518}
}
Almost five years have passed since the release of MobileNetV3.
During this time, the academic community has made significant progress with new network architectures and the application of multi-head attention mechanisms.
Defining the Problem
These new tools invite a practical question: can we put new convolution blocks and attention into MobileNet while keeping both accuracy and latency competitive across different phone chips? FLOPs alone cannot answer it; memory traffic, operator support and the inference runtime also affect latency.
Firstly, recent trends in attention mechanisms have led to two primary approaches:
Hybrid Attention
This approach focuses on combining the strengths of CNNs and Transformers, making attention mechanisms more suitable for lightweight networks.
Notable works include:
- [21.08] Mobile-Former: Bridging MobileNet and Transformer
- [21.10] MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
- [23.03] FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization
Efficient Attention
This approach improves MHSA efficiency by reducing the number of key/value tokens or using linear attention. Both can reduce cost, but not every method below changes quadratic complexity to linear complexity.
Key papers include:
- [21.07] CMT: Convolutional Neural Networks Meet Vision Transformers
- [22.05] EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction
- [22.06] Separable Self-attention for Mobile Vision Transformers
- [22.07] Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios
- [22.12] Rethinking Vision Transformers for MobileNet Size and Speed
Various Convolutional Network Architectures
Beyond attention mechanisms, there are other architecture design studies worth considering:
These serve as valuable references.
Solving the Problem
Network Architecture
The authors first extended the inverted bottleneck (IB) from MobileNet-V2.
In the diagram, the Inverted Bottleneck is the basic structure of MobileNet-V2, which you are likely familiar with.
Look at UIB on the left. Its 1×1 channel expansion and projection remain; the depthwise convolution before expansion and the one after expansion are optional independently. NAS selects their presence and kernel sizes, trading spatial mixing against cost.
This enables four different combinations:
- Both selected: Extra DW
- First DW selected: spatial mixing before expansion, yielding a ConvNeXt-like structure without reproducing all of its normalization and activation choices
- Middle DW selected: Original IB
- Neither selected: two pointwise convolutions with activation and normalization in between, like an FFN
This flexible module is termed UIB (Universal Inverted Bottleneck) in the paper.
The four options are not inserted uniformly throughout the network. The Conv-S configuration below selects ExtraDW where resolution drops, then uses IB and a ConvNeXt-like block at 14×14. In the separate Table 1 searches, the UIB search reaches 83.3% top-1 at 2.68 ms on a Pixel 8 EdgeTPU, while an IB-only search reaches 82.3% at 2.61 ms. This supports an accuracy gain in that setting; latency does not improve at the same time.
Mobile MQA
After designing the convolutional network architecture, the authors introduced Mobile MQA, a module based on Multi-Query Attention (MQA).
This structure originates from:
If you recall the basic structure of attention mechanisms, they consist of Query, Key, and Value.
In multi-head attention, each head has its own Query, Key, and Value.
The main idea here is that each head has a different Query but shares the same Key and Value.
Sharing keys and values reduces projection cost and memory bandwidth demand. Accuracy is evaluated empirically. The design is especially useful when the number of tokens in a batch is small relative to feature dimensionality.
Mobile inference often uses batch size 1, with attention in low-resolution, high-channel-count stages, making this a relevant operating regime.
Unlike text-based attention mechanisms, this one is applied to images.
Thus, the authors also referenced another paper:
Spatial-Reduction Attention (SRA) reduces key/value resolution while retaining query resolution.
Mobile MQA combines shared keys and values with asymmetric spatial reduction. Queries retain their resolution; keys and values use a stride-2 3×3 depthwise convolution where reduction is needed and an identity mapping elsewhere. In the paper's Table 3 experiment at a 16×16 stage, queries keep 256 positions while keys and values fall to 8×8, or 64 positions. S23 GPU latency changes from 7.4 to 5.9 ms and top-1 from 80.77% to 80.71%, a 0.06-point drop. This whole-model Hybrid-M comparison is distinct from the MHSA/MQA ablation in Table 2 below. Calling the method simply “replacing PVT pooling with convolution” is misleading because PVT versions also differ in their spatial reduction.
Cross-Device Optimization
The core goal is a useful accuracy/latency trade-off across devices. NAS first selects coarse widths and convolution choices, then searches whether to use each optional UIB depthwise convolution and which kernel size to choose. This makes candidate architectures easier to evaluate than searching all options in one pass.
During the experiments, the authors gathered several key insights:
- Multi-path efficiency issues:
- Group convolutions and similar multi-path designs, despite having lower FLOPs counts, can be less efficient because of memory access complexity.
- Hardware support is crucial:
- Advanced modules like Squeeze and Excite (SE), GELU, and LayerNorm were poorly supported on the DSPs evaluated in the paper.
- LayerNorm lags behind BatchNorm under those deployment conditions.
- SE can also be a latency bottleneck on accelerators; these observations are specific to the tested hardware and software.
- The power of simplicity:
- Simple modules like ReLU, Conv2D, DW-Conv2D, and BatchNorm are well-supported on mobile devices.
The NAS search space therefore favors widely supported standard operators and UIB. The authors first select Conv backbones, then interleave Mobile MQA in later stages to build Hybrid variants. Attention is not built into every UIB.
Discussion
ImageNet Experimental Results
- means no measurement and Failed means the platform did not support that model.MNv4 models mostly lie on or near the accuracy/latency Pareto frontier of the tested hardware. Compatibility still has limits: Hybrid models were incompatible with the tested DSP, and the CoreML chart excludes Hybrid models because a corresponding PyTorch MQA implementation was unavailable. The CPU, DSP and EdgeTPU measurements use INT8 TFLite, mobile GPUs use FP16, and the iPhone 13 uses FP16 CoreML. Absolute milliseconds across devices are not measurements under one common runtime.
- Named models on the same device: MNv4-Conv-S scores 73.8% top-1 and takes 2.4 ms on the Pixel 6 CPU. Retrained MobileNetV2 scores 73.4% and takes 5.0 ms. At similar accuracy, MNv4 is about 2.1 times faster on that CPU, while both take 0.7 ms on the Pixel 8 EdgeTPU.
- Medium-sized models: MNv4-Conv-M scores 79.9% and takes 11.4 ms on Pixel 6 CPU, 4.1 ms on S23 GPU and 1.1 ms on iPhone 13 CoreML. Similarly accurate FastViT-S12 scores 79.8% and takes 83.0, 47.0 and 1.6 ms, respectively. Ratios depend on the particular model and runtime.
- Missing comparisons: Hybrid-M scores 80.7%, but Table 6 marks its Pixel 4 DSP entry
Failedand iPhone 13 CoreML entry-. FastViT's DSP entries also sayFailed. EfficientViT has S23 GPU and CoreML measurements but no CPU or EdgeTPU results in this table. Results from other devices cannot fill these gaps or prove every Hybrid variant works everywhere.
The authors note that MNv4's success is due to the use of UIB blocks, enhanced NAS recipes, and carefully designed search spaces.
Its value lies in using measurements across platforms during design, rather than guaranteeing optimal performance on every device. The table reports a median statistic over repeated runs on each device, not a prediction for untested phones.
Effectiveness of MQA
Experiments confirmed the advantages of MQA in hybrid models.
The percentages concern three added attention blocks after subtracting the Conv-L base model. Against MHSA, MQA reduces attention-only latency by 84.2%, 39.0% and 41.1% on Pixel 7 EdgeTPU, Pixel 8 EdgeTPU and S23 GPU, respectively. Top-1 changes from 85.27% to 85.24%, a 0.03-point drop.
The 28.6% MAC and 25.5% parameter reductions likewise concern only the added attention component. For the complete model, Table 2 lists 6.7G→6.5G MACs and 36.0M→34.7M parameters. On the S23 GPU, whole-model latency falls from 16.46 to 15.10 ms, a reduction of about 8.3%. Mobile MQA also rearranges Einsum tensor indices to avoid unnecessary transposes; that implementation optimization should not be conflated with the isolated effect of sharing keys and values.
MNv4-Conv-S
The five named models in the main results table are MNv4-Conv-S, Conv-M, Conv-L, Hybrid-M, and Hybrid-L. MNv4-Conv-S is the smallest. The table below describes only this Conv model with a 224×224 input; it is not the layer-by-layer specification of the other four.
For other model architectures, please refer to the paper directly.
DW K1/K2 are the two optional UIB positions described above.Conclusion
This model family brings measurements across several mobile hardware platforms into the design process.
UIB lets the search choose two sites for spatial mixing, while Mobile MQA uses shared keys and values and optional spatial reduction in later stages. With NAS and measurements across devices, several MNv4 configurations approach the Pareto frontier of the evaluated hardware. The Conv/Hybrid variant, training data and platform support still matter.
*
In past development experiences, every new lightweight model required extensive testing on various phones...
- Not to see which is best, but to see which phones have issues.
For example, the FastViT architecture drew attention in 2023, yet its latency away from the iPhone is striking. Table 6 offers a checkable example: FastViT-S12 takes 1.6 ms on iPhone 13 CoreML and 83.0 ms on Pixel 6 CPU; MNv4-Conv-M, at about the same 79.9% top-1, takes 1.1 and 11.4 ms. Absolute times across devices are not directly comparable, but candidate rankings within each device are worth checking.
We appreciate Google testing this round for us, giving us a useful starting point. Deployment still requires testing on our own target devices and runtimes.