Back to paper library
arXiv 2022CVPR 2023ReparameterizationPaper notes

MobileOne

One Millisecond Optimization

Original paper

MobileOne: An Improved One millisecond Mobile Backbone

Paper authors

Pavan Kumar Anasosalu Vasu · James Gabriel · Jeff Zhu · Oncel Tuzel · Anurag Ranjan

Publication
CVPR 2023Conference
First submitted to arXiv
arXiv:2206.04040
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{reparameterizationmobileone2022,
  title = {{MobileOne: An Improved One millisecond Mobile Backbone}},
  author = {Pavan Kumar Anasosalu Vasu and James Gabriel and Jeff Zhu and Oncel Tuzel and Anurag Ranjan},
  year = {2022},
  eprint = {2206.04040},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.04040}
}
On this page

The concept of reparameterization gained attention around 2021 after the introduction of RepVGG.

Researchers have been exploring ways to find suitable reparameterization methods in model architecture to enhance performance.

Problem Definition

Inspired by the success of RepVGG, the authors aimed to explore the effects of reparameterization by designing another structural unit for reparameterization and conducted neural network deployment and benchmarking on an iPhone 12 using CoreML.

Like RepVGG, MobileOne adds mergeable linear/affine branches during training and folds them together for inference. The network still contains ReLU and is not a linear model. MobileOne also relaxes regularization, including weight decay, during training to avoid over-regularizing small models.

Problem Solving

Bottleneck Analysis - Activation Functions

Paper Table 2; source: arXiv:2206.04040v2

Table: Paper Table 2. Only the activation changes: ReLU takes 1.53 ms, GELU 1.63 ms, and Dynamic Shift-Max 57.04 ms.

To analyze the impact of activation functions on latency, the authors built a 30-layer convolutional neural network and conducted benchmarking on an iPhone 12 using different activation functions.

The table above shows that despite having identical architectures except for the activation functions, the latencies varied significantly.

These latency differences are mainly due to recently introduced activation functions. Although these functions showed significant accuracy improvements in ultra-low FLOPs models like MicroNet, their usage can incur high latency costs.

Therefore, to reduce latency costs, the authors use ReLU in most MobileOne blocks. The S4 configuration still uses SE-ReLU in its final two stages.

Bottleneck Analysis - Architecture

Paper Table 3; source: arXiv:2206.04040v2

Table: Paper Table 3. The baseline takes 1.53 ms; adding SE or skip connections separately gives 2.10 or 2.62 ms. These are not cumulative steps.

Next, they considered two common architectures: skip connections and Squeeze-Excite modules.

Two critical factors affecting runtime performance are memory access costs and parallelism. In multi-branch architectures, memory access costs increase significantly as activations for each branch need to be stored to compute the next tensor in the computation graph. If the number of branches is small, this memory bottleneck can be avoided. Additionally, forced synchronization blocks (e.g., global pool operations used in Squeeze-Excite blocks) can also impact overall runtime due to synchronization costs.

To expose memory access and synchronization costs, the authors used the 30-layer network without either module as the baseline, then added skip connections or Squeeze-Excite separately.

Thus, MobileOne adopts an inference architecture with its mergeable branches removed, resulting in smaller memory access costs. Squeeze-Excite modules remain in selected blocks of the largest S4 variant to improve accuracy.

Model Architecture

Paper Figure 3; source: arXiv:2206.04040v2
Figure: Paper Figure 3. kk counts replicated convolution branches at training time; each convolution type is fused for inference, while the activation remains.

RepVGG adds mergeable branches to VGG-style 3×3 convolutions; it does not turn arbitrary ResNet residual blocks into single convolutions. MobileOne instead targets MobileNetV1-style depthwise/pointwise blocks.

As shown in the above figure, this time the reparameterization target is the form of depthwise separable convolutions, first performing a 3x3 depthwise convolution, followed by a 1x1 pointwise convolution. Then, reparameterizable skip connections with batch normalization and replicated structure branches are introduced.

At inference, the mergeable additive branches become a single convolution. The SE modules retained in S4 are not among those mergeable branches. For a convolution layer with a kernel size of K, input channel dimension CinC_{in}, and output channel dimension CoutC_{out}, the weight matrix is represented as W′∈RCout×(Cin/g)×K×KW' \in \mathbb{R}^{C_{out} \times (C_{in}/g) \times K \times K}, and bias as b′∈RCoutb' \in \mathbb{R}^{C_{out}}, where gg is the group count (equal to input channels for depthwise convolution). BN contains accumulated mean μ\mu, variance σ2\sigma^2, scale γ\gamma, and offset β\beta.

First fold BN using fixed statistics, pad smaller kernels to the same size, and represent identity branches as convolutions. Stride, groups and spatial alignment must match. Then, the convolution layer's weight WW and bias bb during inference are calculated as follows: weight WW equals the sum of weights WiW_{i} from all branches, i.e., W=∑i=1MWiW = \sum_{i=1}^{M} W_{i}; bias bb equals the sum of biases bib_{i} from all branches, i.e., b=∑i=1Mbib = \sum_{i=1}^{M} b_{i}, where MM is the number of branches.

Model Configuration

Paper Table 7; source: arXiv:2206.04040v2

Table: Paper Table 7. In each tuple, α\alpha scales width and kk counts training branches; S0 often uses k=4k=4, other variants mainly use k=1k=1, and S4 adds SE-ReLU in later stages.

Recent studies scale model width, depth, and resolution. This table lists five S0–S4 configurations that also differ in training branch count and SE placement. The paper does not increase input resolution in this comparison.

MobileOne scales depth similarly to MobileNet-V2, using shallower early stages at higher feature-map resolutions as these layers are much slower than later stages.

Since MobileOne has no multi-branch architecture during inference, it avoids the data movement costs associated with such architectures. This allows for model parameter expansion without significant latency cost increases, which is a major advantage compared to competitive multi-branch architectures like MobileNet-V2 and EfficientNets.

Discussion

ImageNet Performance

Paper Figure 1; source: arXiv:2206.04040v2
Figure: Paper Figure 1. The left panel plots iPhone 12 latency against ImageNet top-1; the bottom-right panel plots COCO mAP against classification accuracy, with bubble size indicating phone latency. Distilled models are orange.
Paper Table 9; source: arXiv:2206.04040v2

Table: Paper Table 9. Classification models use 224 × 224 inputs without distillation. MobileOne-S0/S1 reach 0.79/0.89 ms on the phone; S4 takes 1.86 ms.

The authors evaluated MobileOne models on the ImageNet dataset, which contains 1.28 million training images and 50,000 validation images from 1,000 classes.

All MobileOne variants were trained from scratch using PyTorch on machines equipped with 8 NVIDIA GPUs; the comparison models in the table do not necessarily share this recipe. MobileOne training used SGD with momentum for 300 epochs and an effective batch size of 256. It also used cross-entropy loss with label smoothing of 0.1.

The initial learning rate was set to 0.1 and decayed using a cosine annealing schedule, with the initial weight decay coefficient set to 10−410^{-4}, gradually decaying to 10−510^{-5}.

Larger MobileOne variants (S2, S3, and S4) were trained with AutoAugment, while smaller variants (S0 and S1) used standard data augmentation techniques such as random resized cropping and horizontal flipping.

All versions of MobileOne used EMA (Exponential Moving Average) weights during training with a decay constant of 0.9995.

The main comparison uses 224 × 224 inputs and models trained without distillation. All MobileOne variants have fewer than 20 million parameters, although some additional RepVGG baselines in the table are larger. FLOPs counts were reported using the fvcore library.

In the paper's iPhone 12/CoreML setup, every listed Transformer variant exceeds 4 ms on the phone. Mobile numbers are minimum full round-trip latency after preloading and warm-up; CPU and GPU results use different hardware and statistics. The comparison also depends on the conversion tools and runtime used at the time.

The paper's prose separately cites a MobileFormer variant at 79.3%/70.76 ms and compares it with MobileOne-S4 at 79.4%/1.86 ms, a roughly 38-fold phone-latency ratio; 70.76 ms is not a Mobileformer-52/96 row in Table 9. In Table 9, MobileOne-S3 is 1 percentage point more accurate than EfficientNet-B0, while iPhone 12 latency falls from 1.72 to 1.53 ms, about 11% lower rather than eleven times faster.

Table 9 also reports CPU and GPU latency. Whether MobileOne leads depends on the particular accuracy-matched pair and hardware; not every variant is the fastest on every platform.

Conclusion

The study analyzes efficiency bottlenecks in components used by modern CNNs and shows how reparameterizable structures can ease optimization during training. MobileOne uses mergeable branches to help train its models, then folds them before inference.

With its scaling strategy, MobileOne reaches strong accuracy while remaining efficient on the reported phone and desktop-CPU benchmarks. In the paper's iPhone 12 setting, S0 and S1 run under 1 ms; this does not apply to every variant or device.

The results improve the reported accuracy–latency trade-off and offer a useful starting point for future work and applications. The effect on generalization and speed still needs to be checked on the intended data and hardware.