EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Publication
- ICML 2019Conference
- First submitted to arXiv
- arXiv:1905.11946
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{classiccnnsefficientnet2019,
title = {{EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks}},
author = {Mingxing Tan and Quoc V. Le},
year = {2019},
eprint = {1905.11946},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1905.11946}
}
Common methods to enhance the performance of convolutional neural networks (CNNs) include:
- Deeper Networks: Increasing network depth, such as with ResNet.
- Wider Networks: Increasing network width, like WideResNet.
- Higher Resolution: Increasing the input image resolution.
I add a little here, you add a little there, and each leaderboard score inches upward. Sounds like a pleasant way to spend our time, doesn't it?
Defining the Problem
The paper's authors object: why tune one dimension at a time when the three draw on the same compute budget? They propose Compound Scaling.
Solving the Problem
Two Key Observations
In this paper, the authors present two critical observations that form the foundation of their compound scaling method:
-
Observation 1: Any single dimension scaling improves accuracy, but the improvement saturates as the model size increases.
Paper Figure 3: the x-axes count billions of FLOPs and the y-axes show ImageNet top-1 accuracy. Labels mark the coefficient varied in each panel; later gains flatten. Points across the panels do not all use identical FLOPs. Increasing any one dimension initially raises accuracy, but the curves flatten. In the middle panel, moving from depth coefficient to leaves accuracy near 80% while FLOPs still rise. Continuing to enlarge a single dimension need not be the best use of the next compute budget.
-
Observation 2: Balancing network width, depth, and resolution is crucial for effective model scaling.
Paper Figure 4: all four lines increase width, starting from depth or and resolution multiplier or . The red line uses both the deeper and higher-resolution baseline; the x-axis counts FLOPs, not parameters. This plot varies width from four combinations of starting depth and resolution. Widening the blue baseline saturates early, while the deeper, higher-resolution red baseline achieves better accuracy over the plotted FLOPs range. This supports balancing the dimensions in these experiments.
Compound Scaling
Here, the problem of model scaling is redefined:
Each layer of a CNN can be defined as a function , where is the operator, is the output tensor, and is the input tensor. The entire CNN can be represented as a combination of these layers: . Typically, the layers in a CNN are divided into stages, with all layers in a stage sharing the same architecture.
Therefore, a CNN can be defined as:
Here means function composition, not element-wise multiplication; weight sharing is a separate architectural choice.
The term denotes the layer being repeated times in the -th stage.
The goal of model scaling is to maximize accuracy within given resource constraints. Specifically, we aim to optimize the model by adjusting its depth (), width (), and resolution () while keeping the basic network architecture unchanged.
This can be formalized as an optimization problem:
Subject to:
The authors propose a compound scaling method using a compound coefficient to uniformly scale the network's width, depth, and resolution:
Subject to , where , , and are constants determined by a small grid search, and is a user-specified coefficient controlling the scaling resources.
For standard convolutions, compute scales approximately as , so increasing by 1 roughly doubles FLOPs; actual widths and layer counts require rounding.
Base Model Architecture
To develop an effective baseline network, the authors leveraged Multi-Objective Neural Architecture Search (NAS) to optimize both accuracy and FLOPs, meaning operations per forward pass rather than operations per second.
They used as the optimization objective, where and represent the accuracy and FLOPs of model , respectively. is the target FLOPs, and is the reward exponent (distinct from the earlier width multiplier) and a hyperparameter controlling the balance between accuracy and FLOPs.
The search yielded EfficientNet-B0, whose architecture is similar to MnasNet but slightly larger, targeting 400M FLOPs.
The table shows EfficientNet-B0 by stage, rather than listing every repeated MBConv layer separately:
EfficientNet-B0 mainly uses mobile inverted bottleneck convolutions (MBConv) with squeeze-and-excitation. In Table 1, the 1 or 6 in MBConv1/MBConv6 is the expansion ratio, 3×3 or 5×5 is the kernel size, and the rightmost column gives the number of layers in the stage.
Model Scaling Architecture
Starting from the baseline EfficientNet-B0, the authors applied the compound scaling method in two steps to scale the network:
-
Step 1: First, fix , assuming twice the resources are available, and perform a small-scale grid search based on equations 2 and 3 to determine the optimal values for , , and . For EfficientNet-B0, the optimal values were , , and , satisfying .
-
Step 2: Then, fix , , and as constants and use different values to scale the baseline network, resulting in EfficientNet-B1 to B7.
Notably, while it is possible to search for , , and directly on large models for better performance, the search cost becomes prohibitively expensive on large models. The authors' method effectively addresses this issue by performing the search once on a small baseline network (Step 1) and then scaling all other models using the same scaling coefficients (Step 2).
Discussion
Importance of Compound Scaling
All models start from EfficientNet-B0. The macaron and maze examples let us inspect where each model places its activation:
- Single-dimension variants: in these two images, some hot regions cover only part of the object or shape.
- Compound-scaled variant: the hot regions cover more class-relevant details. This is a visual observation; the classification results below are needed to assess the overall gain.
Additionally, the authors observed the performance of several scaling methods:
-
Single-Dimension Scaling Methods:
- Depth Scaling: Increasing the network depth, i.e., the number of layers.
- Width Scaling: Increasing the network width, i.e., the number of channels per layer.
- Resolution Scaling: Increasing the input image resolution.
-
Compound Scaling Method:
- Combining depth, width, and resolution growth, coordinated according to the proposed , , parameters.
Figure 8 below compares scaling methods from the same EfficientNet-B0 baseline. Unpictured Table 7 lists configurations at about 1.8–1.9B FLOPs: depth, width and resolution alone reach 79.0/78.9/79.1% top-1, whereas compound scaling reaches 81.1%, a 2.0–2.2 percentage-point gap. The paper's “up to 2.5 points” refers to another comparison along the curves, not this Table 7 group.
Results on ImageNet
In this paper, the authors used the following specific training configurations:
- Optimizer: RMSProp, decay and momentum both 0.9; BatchNorm momentum 0.99
- Weight Decay:
- Initial Learning Rate: 0.256, decaying by 0.97 every 2.4 epochs
- Activation Function: Swish (the form used here is equivalent to SiLU)
- Data Augmentation: AutoAugment
- Regularization: stochastic depth with path survival probability 0.8
The authors increased dropout linearly from 0.2 for B0 to 0.5 for B7. They also set aside 25k training images for early stopping and reported results on the original validation set. Dropout and stochastic depth are distinct operations.
The savings vary by comparator. B7 reaches 84.3% top-1 with 66M parameters and 37B FLOPs; GPipe also reaches 84.3% but has 557M parameters, about 8.4 times as many. Table 2 omits GPipe's FLOPs.
For B3 versus ResNeXt-101, Table 2 reports 81.6/80.9% top-1 and 1.8/32B FLOPs, about an 18-fold operation-count difference in these paper results. It is not a latency ratio for every device.
Inference Speed
To validate latency, the authors also measured the inference latency of some representative CNNs on a single Intel Xeon E5-2690 CPU core with batch size 1.
The table below shows the average latency of 20 runs:
Across 20 runs in that setup, B1 is about 5.7 times faster than ResNet-152 and B7 about 6.1 times faster than GPipe. These are one-core Xeon, batch-one results; GPU and mobile latency require new measurements. Table 4's accuracy values do not fully match Table 2, so they should not be presented as identical evaluation results.
Parameter-Accuracy Curve
Along the red curve from B0 to B7, accuracy grows with model scale. Compared with some plotted alternatives, EfficientNet reaches similar or higher top-1 accuracy at lower parameter count or FLOPs. The two x-axes differ, and latency requires separate measurement. In the right inset, NASNet-A is 80.7%, while Table 2 lists 82.7%; the paper does not explain this discrepancy here, so the main ImageNet comparison above follows Table 2.
Conclusion
The depth, width, and resolution of a model are crucial factors influencing the performance of convolutional neural networks.
The paper proposes compound scaling: coordinate changes to network depth, width and resolution to improve accuracy and efficiency. Table 7 and Figure 8 support this strategy when the scaling methods start from the same B0 baseline. B0 itself was found by NAS and the training setup was tuned, so not all of B7's advantage over older models can be assigned to the scaling equation.
Future work could refine the choice of compound-scaling parameters and explore whether the method helps other kinds of neural networks.