EfficientNetV2: Smaller Models and Faster Training
- Publication
- ICML 2021Conference
- First submitted to arXiv
- arXiv:2104.00298
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{classiccnnsefficientnetv22021,
title = {{EfficientNetV2: Smaller Models and Faster Training}},
author = {Mingxing Tan and Quoc V. Le},
year = {2021},
eprint = {2104.00298},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2104.00298}
}
After the first EfficientNet, its authors thought there was still room to improve.
They proposed EfficientNetV2, a smaller model that trains faster while remaining efficient.
Problem Definition
The authors identified three problems with the first generation:
- Large first-generation models use high-resolution images that consume substantial memory and constrain training batch size.
- Depthwise convolutions in early network stages can run slowly on an accelerator despite their low FLOPs.
- The original scaling rule applies the same depth multiplier to every stage, even though the stages have different compute costs.
Why might fewer operations take longer? The authors point to earlier work, Accelerator-aware Neural Network Design using AutoML:
Now that the problems are clear, it is time to change the design.
Solution
Model Architecture
First compare the two blocks in the EfficientNetV2 paper. MBConv expands channels with a 1×1 convolution and then applies a 3×3 depthwise convolution. Fused-MBConv replaces those first two operations with one standard 3×3 convolution.
Depthwise convolution saves parameters and FLOPs, but it may underuse a particular accelerator. The authors include the previously introduced Fused-MBConv in architecture search. The resulting design uses fused blocks in early stages and retains MBConv later, with placement determined by throughput and compute cost.
Progressive Learning
Earlier methods changed image size during training, but sometimes lost accuracy. The authors suspect that regularization stayed fixed when it should have changed with image size. Larger models often need stronger regularization: EfficientNet-B7, for example, uses more dropout and stronger augmentation than B0. The authors extend that idea to a fixed network: they hypothesize that smaller training images need weaker regularization, and larger images need stronger regularization. Its parameter count does not change with image size.
They first hold the model fixed and vary image size and RandAugment magnitude:
The best mean accuracies are 78.3%, 81.6%, and 83.2%. One augmentation strength is not optimal for every image size. Progressive learning therefore raises image size alongside RandAugment, mixup, and dropout in stages. For EfficientNetV2-S, training image size grows from 128 to 300 pixels; the regularization schedule also varies by model.
Another Round of NAS
After changing the blocks, the authors run neural architecture search again. The search varies MBConv versus Fused-MBConv, 3×3 versus 5×5 kernels, expansion ratios of 1, 4, or 6, and the number of layers per stage. About 1,000 candidates are evaluated after roughly 10 epochs of training with smaller images. The reward considers accuracy, training step time, and parameter count, so this is not a search for the fewest FLOPs alone.
The original EfficientNet greatly increased inference resolution while scaling up, which also raised training memory cost. EfficientNetV2 caps the maximum inference image size at 480 pixels and places more added layers in later stages. Its progressive training schedule controls training resolution separately.
Discussion
Where to Place Fused-MBConv?
Replacing every stage with Fused-MBConv raises parameters from 19.3M with no fused stages to 132M, and FLOPs from 4.5B to 34.4B; top-1 accuracy falls from 82.8% to 81.7%. With fusion only in stages 1–3, accuracy is 83.1%, TPUv3 throughput rises from 262 to 362 images per second per core, and V100 throughput rises from 155 to 216.
On the V100, however, stages 1–5 reach 223 images per second, slightly above the 216 for stages 1–3, at the cost of more parameters and FLOPs. Placement depends on both hardware and model cost.
Network Search Results
Stage 1 uses a fused block with expansion ratio 1; stages 2 and 3 use ratio 4. Stage 4 uses MBConv4, and stages 5 and 6 use MBConv6. The search favors 3×3 kernels in most blocks and allocates more layers to later stages. The actual model is more specific than “replace every convolution with a faster one.”
Speeding Up Training
At about 85% accuracy, the EfficientNetV2 curve lies to the left of the reproduced first-generation EfficientNet curve. That supports a training-efficiency advantage in this setup. The next table reports inference latency separately.
Results on ImageNet
EfficientNetV2-M reaches 85.1% top-1 versus 84.7% for EfficientNet-B7. Their training times are 13 and 139 hours on the paper's 32-TPU-core basis, or approximately an elevenfold difference. Their V100 inference times are reported separately as 57 and 170 ms.
EfficientNetV2-L trained directly on ImageNet reaches 85.7%, above the table’s ViT-L/16 (21k) at 85.3%. Their pretraining conditions differ. For a comparison in which both models have ImageNet21k pretraining, use EfficientNetV2-L (21k) at 86.8%.
In that (21k) comparison, EfficientNetV2-L exceeds ViT-L/16 by 1.5 percentage points, with 120M versus 304M parameters and 53B versus 192B FLOPs. Training takes 26 versus 172 hours, about 6.6 times less time; V100 inference takes 98 versus 195 ms, about twice as fast. The RegNet and ResNeSt rows have no training-time data.
The authors also observe that once their models approach 85% top-1, more pretraining data helps more than simply increasing model size. ImageNet21k contains about 13 million images; EfficientNetV2 pretraining on it takes about two days with 32 TPU cores in the reported setup. The 85% point is an observation from these experiments, not a general threshold for other datasets.
Conclusion
For those of us who regularly build models, this paper offers two useful ideas:
- Using
Fused-MBConvin the first three stages can improve this model's performance; check its placement against throughput, parameters and FLOPs on the target hardware. - As image size changes during training, adjust regularization strength instead of keeping it fixed.
These ideas can help us train models more efficiently for practical work. Their speedup in a particular application still needs to be measured on its own hardware and data.