Back to paper library
arXiv 2024ECCV 2024Efficient modelsPaper notes

MobileNet-V4

Five Years of Evolution

Original paper

MobileNetV4 -- Universal Models for the Mobile Ecosystem

Paper authors

Danfeng Qin · Chas Leichner · Manolis Delakis …

Show all 14 authors

Danfeng Qin · Chas Leichner · Manolis Delakis · Marco Fornoni · Shixin Luo · Fan Yang · Weijun Wang · Colby Banbury · Chengxi Ye · Berkin Akin · Vaibhav Aggarwal · Tenghui Zhu · Daniele Moro · Andrew Howard

Publication
ECCV 2024Conference
First submitted to arXiv
arXiv:2404.10518
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{lightweightmobilenetv42024,
  title = {{MobileNetV4 -- Universal Models for the Mobile Ecosystem}},
  author = {Danfeng Qin and Chas Leichner and Manolis Delakis and Marco Fornoni and Shixin Luo and Fan Yang and Weijun Wang and Colby Banbury and Chengxi Ye and Berkin Akin and Vaibhav Aggarwal and Tenghui Zhu and Daniele Moro and Andrew Howard},
  year = {2024},
  eprint = {2404.10518},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.10518}
}
On this page

Almost five years have passed since the release of MobileNetV3.

During this time, the academic community has made significant progress with new network architectures and the application of multi-head attention mechanisms.

Defining the Problem

These new tools invite a practical question: can we put new convolution blocks and attention into MobileNet while keeping both accuracy and latency competitive across different phone chips? FLOPs alone cannot answer it; memory traffic, operator support and the inference runtime also affect latency.

Firstly, recent trends in attention mechanisms have led to two primary approaches:

Hybrid Attention

This approach focuses on combining the strengths of CNNs and Transformers, making attention mechanisms more suitable for lightweight networks.

Notable works include:

Efficient Attention

This approach improves MHSA efficiency by reducing the number of key/value tokens or using linear attention. Both can reduce cost, but not every method below changes quadratic complexity to linear complexity.

Key papers include:

Various Convolutional Network Architectures

Beyond attention mechanisms, there are other architecture design studies worth considering:

These serve as valuable references.

Solving the Problem

Network Architecture

Paper Figure 4: orange marks depthwise convolutions, blue marks 1×1 pointwise convolutions, and Alternative Fused IB is a separate block on the right.

The authors first extended the inverted bottleneck (IB) from MobileNet-V2.

In the diagram, the Inverted Bottleneck is the basic structure of MobileNet-V2, which you are likely familiar with.

Look at UIB on the left. Its 1×1 channel expansion and projection remain; the depthwise convolution before expansion and the one after expansion are optional independently. NAS selects their presence and kernel sizes, trading spatial mixing against cost.

This enables four different combinations:

  1. Both selected: Extra DW
  2. First DW selected: spatial mixing before expansion, yielding a ConvNeXt-like structure without reproducing all of its normalization and activation choices
  3. Middle DW selected: Original IB
  4. Neither selected: two pointwise convolutions with activation and normalization in between, like an FFN

This flexible module is termed UIB (Universal Inverted Bottleneck) in the paper.

The four options are not inserted uniformly throughout the network. The Conv-S configuration below selects ExtraDW where resolution drops, then uses IB and a ConvNeXt-like block at 14×14. In the separate Table 1 searches, the UIB search reaches 83.3% top-1 at 2.68 ms on a Pixel 8 EdgeTPU, while an IB-only search reaches 82.3% at 2.61 ms. This supports an accuracy gain in that setting; latency does not improve at the same time.

Mobile MQA

After designing the convolutional network architecture, the authors introduced Mobile MQA, a module based on Multi-Query Attention (MQA).

This structure originates from:

If you recall the basic structure of attention mechanisms, they consist of Query, Key, and Value.

In multi-head attention, each head has its own Query, Key, and Value.

The main idea here is that each head has a different Query but shares the same Key and Value.

Sharing keys and values reduces projection cost and memory bandwidth demand. Accuracy is evaluated empirically. The design is especially useful when the number of tokens in a batch is small relative to feature dimensionality.

Mobile inference often uses batch size 1, with attention in low-resolution, high-channel-count stages, making this a relevant operating regime.

Unlike text-based attention mechanisms, this one is applied to images.

Thus, the authors also referenced another paper:

Spatial-Reduction Attention (SRA) reduces key/value resolution while retaining query resolution.

Mobile MQA combines shared keys and values with asymmetric spatial reduction. Queries retain their resolution; keys and values use a stride-2 3×3 depthwise convolution where reduction is needed and an identity mapping elsewhere. In the paper's Table 3 experiment at a 16×16 stage, queries keep 256 positions while keys and values fall to 8×8, or 64 positions. S23 GPU latency changes from 7.4 to 5.9 ms and top-1 from 80.77% to 80.71%, a 0.06-point drop. This whole-model Hybrid-M comparison is distinct from the MHSA/MQA ablation in Table 2 below. Calling the method simply “replacing PVT pooling with convolution” is misleading because PVT versions also differ in their spatial reduction.

Cross-Device Optimization

The core goal is a useful accuracy/latency trade-off across devices. NAS first selects coarse widths and convolution choices, then searches whether to use each optional UIB depthwise convolution and which kernel size to choose. This makes candidate architectures easier to evaluate than searching all options in one pass.

During the experiments, the authors gathered several key insights:

  1. Multi-path efficiency issues:
    • Group convolutions and similar multi-path designs, despite having lower FLOPs counts, can be less efficient because of memory access complexity.
  2. Hardware support is crucial:
    • Advanced modules like Squeeze and Excite (SE), GELU, and LayerNorm were poorly supported on the DSPs evaluated in the paper.
    • LayerNorm lags behind BatchNorm under those deployment conditions.
    • SE can also be a latency bottleneck on accelerators; these observations are specific to the tested hardware and software.
  3. The power of simplicity:
    • Simple modules like ReLU, Conv2D, DW-Conv2D, and BatchNorm are well-supported on mobile devices.

The NAS search space therefore favors widely supported standard operators and UIB. The authors first select Conv backbones, then interleave Mobile MQA in later stages to build Hybrid variants. Attention is not built into every UIB.

Discussion

ImageNet Experimental Results

Paper Figure 1: each panel plots latency on a different device against ImageNet-1k top-1; circles are Conv and squares are Hybrid. Compare models within one panel before drawing conclusions about speed.
Paper Table 6: - means no measurement and Failed means the platform did not support that model.

MNv4 models mostly lie on or near the accuracy/latency Pareto frontier of the tested hardware. Compatibility still has limits: Hybrid models were incompatible with the tested DSP, and the CoreML chart excludes Hybrid models because a corresponding PyTorch MQA implementation was unavailable. The CPU, DSP and EdgeTPU measurements use INT8 TFLite, mobile GPUs use FP16, and the iPhone 13 uses FP16 CoreML. Absolute milliseconds across devices are not measurements under one common runtime.

  1. Named models on the same device: MNv4-Conv-S scores 73.8% top-1 and takes 2.4 ms on the Pixel 6 CPU. Retrained MobileNetV2 scores 73.4% and takes 5.0 ms. At similar accuracy, MNv4 is about 2.1 times faster on that CPU, while both take 0.7 ms on the Pixel 8 EdgeTPU.
  2. Medium-sized models: MNv4-Conv-M scores 79.9% and takes 11.4 ms on Pixel 6 CPU, 4.1 ms on S23 GPU and 1.1 ms on iPhone 13 CoreML. Similarly accurate FastViT-S12 scores 79.8% and takes 83.0, 47.0 and 1.6 ms, respectively. Ratios depend on the particular model and runtime.
  3. Missing comparisons: Hybrid-M scores 80.7%, but Table 6 marks its Pixel 4 DSP entry Failed and iPhone 13 CoreML entry -. FastViT's DSP entries also say Failed. EfficientViT has S23 GPU and CoreML measurements but no CPU or EdgeTPU results in this table. Results from other devices cannot fill these gaps or prove every Hybrid variant works everywhere.

The authors note that MNv4's success is due to the use of UIB blocks, enhanced NAS recipes, and carefully designed search spaces.

Its value lies in using measurements across platforms during design, rather than guaranteeing optimal performance on every device. The table reports a median statistic over repeated runs on each device, not a prediction for untested phones.

Effectiveness of MQA

Paper Table 2: the latency percentages in the last row apply to the added attention blocks after subtracting the base model.

Experiments confirmed the advantages of MQA in hybrid models.

The percentages concern three added attention blocks after subtracting the Conv-L base model. Against MHSA, MQA reduces attention-only latency by 84.2%, 39.0% and 41.1% on Pixel 7 EdgeTPU, Pixel 8 EdgeTPU and S23 GPU, respectively. Top-1 changes from 85.27% to 85.24%, a 0.03-point drop.

The 28.6% MAC and 25.5% parameter reductions likewise concern only the added attention component. For the complete model, Table 2 lists 6.7G→6.5G MACs and 36.0M→34.7M parameters. On the S23 GPU, whole-model latency falls from 16.46 to 15.10 ms, a reduction of about 8.3%. Mobile MQA also rearranges Einsum tensor indices to avoid unnecessary transposes; that implementation optimization should not be conflated with the isolated effect of sharing keys and values.

MNv4-Conv-S

The five named models in the main results table are MNv4-Conv-S, Conv-M, Conv-L, Hybrid-M, and Hybrid-L. MNv4-Conv-S is the smallest. The table below describes only this Conv model with a 224×224 input; it is not the layer-by-layer specification of the other four.

For other model architectures, please refer to the paper directly.

Paper Table 11: a 224²×3 input passes through Conv2D and two FusedIB blocks to 28²×64, then ExtraDW to 14²×96 and eventually 7²×128 before the classifier. DW K1/K2 are the two optional UIB positions described above.

Conclusion

This model family brings measurements across several mobile hardware platforms into the design process.

UIB lets the search choose two sites for spatial mixing, while Mobile MQA uses shared keys and values and optional spatial reduction in later stages. With NAS and measurements across devices, several MNv4 configurations approach the Pareto frontier of the evaluated hardware. The Conv/Hybrid variant, training data and platform support still matter.

*

In past development experiences, every new lightweight model required extensive testing on various phones...

  • Not to see which is best, but to see which phones have issues.

For example, the FastViT architecture drew attention in 2023, yet its latency away from the iPhone is striking. Table 6 offers a checkable example: FastViT-S12 takes 1.6 ms on iPhone 13 CoreML and 83.0 ms on Pixel 6 CPU; MNv4-Conv-M, at about the same 79.9% top-1, takes 1.1 and 11.4 ms. Absolute times across devices are not directly comparable, but candidate rankings within each device are worth checking.

We appreciate Google testing this round for us, giving us a useful starting point. Deployment still requires testing on our own target devices and runtimes.