Back to paper library
arXiv 2019ICML 2019Classic CNNsPaper notes

EfficientNet

Compound Scaling

Original paper

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Paper authors

Mingxing Tan · Quoc V. Le

Publication
ICML 2019Conference
First submitted to arXiv
arXiv:1905.11946
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{classiccnnsefficientnet2019,
  title = {{EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks}},
  author = {Mingxing Tan and Quoc V. Le},
  year = {2019},
  eprint = {1905.11946},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.11946}
}
On this page

Common methods to enhance the performance of convolutional neural networks (CNNs) include:

  1. Deeper Networks: Increasing network depth, such as with ResNet.
  2. Wider Networks: Increasing network width, like WideResNet.
  3. Higher Resolution: Increasing the input image resolution.

I add a little here, you add a little there, and each leaderboard score inches upward. Sounds like a pleasant way to spend our time, doesn't it?

Defining the Problem

The paper's authors object: why tune one dimension at a time when the three draw on the same compute budget? They propose Compound Scaling.

Paper Figure 2: (a) is a baseline; (b)–(d) each enlarge one dimension, while (e) enlarges all three. The bars are schematic, not a layer-by-layer specification of EfficientNet-B0.

Solving the Problem

Two Key Observations

In this paper, the authors present two critical observations that form the foundation of their compound scaling method:

  • Observation 1: Any single dimension scaling improves accuracy, but the improvement saturates as the model size increases.

    Paper Figure 3: the x-axes count billions of FLOPs and the y-axes show ImageNet top-1 accuracy. Labels mark the coefficient varied in each panel; later gains flatten. Points across the panels do not all use identical FLOPs.

    Increasing any one dimension initially raises accuracy, but the curves flatten. In the middle panel, moving from depth coefficient d=6d=6 to 88 leaves accuracy near 80% while FLOPs still rise. Continuing to enlarge a single dimension need not be the best use of the next compute budget.

  • Observation 2: Balancing network width, depth, and resolution is crucial for effective model scaling.

    Paper Figure 4: all four lines increase width, starting from depth d=1d=1 or 22 and resolution multiplier r=1r=1 or 1.31.3. The red line uses both the deeper and higher-resolution baseline; the x-axis counts FLOPs, not parameters.

    This plot varies width from four combinations of starting depth and resolution. Widening the blue d=r=1d=r=1 baseline saturates early, while the deeper, higher-resolution red baseline achieves better accuracy over the plotted FLOPs range. This supports balancing the dimensions in these experiments.

Compound Scaling

Here, the problem of model scaling is redefined:

Each layer of a CNN can be defined as a function Yi=Fi(Xi)Y_i = F_i(X_i), where FiF_i is the operator, YiY_i is the output tensor, and XiX_i is the input tensor. The entire CNN NN can be represented as a combination of these layers: N=Fk⊙...⊙F2⊙F1(X1)N = F_k \odot ... \odot F_2 \odot F_1(X_1). Typically, the layers in a CNN are divided into stages, with all layers in a stage sharing the same architecture.

Therefore, a CNN can be defined as:

N=⊙i=1sFiLi(X)N = \odot_{i=1}^s F_i^{L_i}(X)

Here ⊙\odot means function composition, not element-wise multiplication; weight sharing is a separate architectural choice.

The term FiLiF_i^{L_i} denotes the layer FiF_i being repeated LiL_i times in the ii-th stage.

The goal of model scaling is to maximize accuracy within given resource constraints. Specifically, we aim to optimize the model by adjusting its depth (dd), width (ww), and resolution (rr) while keeping the basic network architecture FiF_i unchanged.

This can be formalized as an optimization problem:

  • max⁡d,w,rAccuracy(N(d,w,r))\max_{d,w,r} \text{Accuracy}(N(d, w, r))

Subject to:

  • Memory(N)≤target memory\text{Memory}(N) \leq \text{target memory}
  • FLOPs(N)≤target flops\text{FLOPs}(N) \leq \text{target flops}

The authors propose a compound scaling method using a compound coefficient ϕ\phi to uniformly scale the network's width, depth, and resolution:

  • depth: d=αϕ\text{depth: } d = \alpha^\phi
  • width: w=βϕ\text{width: } w = \beta^\phi
  • resolution: r=γϕ\text{resolution: } r = \gamma^\phi

Subject to α⋅β2⋅γ2≈2\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2, where α\alpha, β\beta, and γ\gamma are constants determined by a small grid search, and ϕ\phi is a user-specified coefficient controlling the scaling resources.

For standard convolutions, compute scales approximately as dw2r2d w^2 r^2, so increasing ϕ\phi by 1 roughly doubles FLOPs; actual widths and layer counts require rounding.

Base Model Architecture

To develop an effective baseline network, the authors leveraged Multi-Objective Neural Architecture Search (NAS) to optimize both accuracy and FLOPs, meaning operations per forward pass rather than operations per second.

They used Acc(m)×(FLOPs(m)T)w\text{Acc}(m) \times \left(\frac{\text{FLOPs}(m)}{T}\right)^w as the optimization objective, where Acc(m)\text{Acc}(m) and FLOPs(m)\text{FLOPs}(m) represent the accuracy and FLOPs of model mm, respectively. TT is the target FLOPs, and w=−0.07w = -0.07 is the reward exponent (distinct from the earlier width multiplier) and a hyperparameter controlling the balance between accuracy and FLOPs.

The search yielded EfficientNet-B0, whose architecture is similar to MnasNet but slightly larger, targeting 400M FLOPs.

The table shows EfficientNet-B0 by stage, rather than listing every repeated MBConv layer separately:

Paper Table 1: each row is one stage. For example, stage 4 starts at 56×56 and repeats an MBConv6 with a 5×5 kernel twice, producing 40 channels. The Resolution column gives the stage input size.

EfficientNet-B0 mainly uses mobile inverted bottleneck convolutions (MBConv) with squeeze-and-excitation. In Table 1, the 1 or 6 in MBConv1/MBConv6 is the expansion ratio, 3×3 or 5×5 is the kernel size, and the rightmost column gives the number of layers in the stage.

Model Scaling Architecture

Starting from the baseline EfficientNet-B0, the authors applied the compound scaling method in two steps to scale the network:

  • Step 1: First, fix ϕ=1\phi = 1, assuming twice the resources are available, and perform a small-scale grid search based on equations 2 and 3 to determine the optimal values for α\alpha, β\beta, and γ\gamma. For EfficientNet-B0, the optimal values were α=1.2\alpha = 1.2, β=1.1\beta = 1.1, and γ=1.15\gamma = 1.15, satisfying α⋅β2⋅γ2≈2\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2.

  • Step 2: Then, fix α\alpha, β\beta, and γ\gamma as constants and use different ϕ\phi values to scale the baseline network, resulting in EfficientNet-B1 to B7.

Notably, while it is possible to search for α\alpha, β\beta, and γ\gamma directly on large models for better performance, the search cost becomes prohibitively expensive on large models. The authors' method effectively addresses this issue by performing the search once on a small baseline network (Step 1) and then scaling all other models using the same scaling coefficients (Step 2).

Discussion

Importance of Compound Scaling

Paper Figure 7: each row has an original image, the same EfficientNet-B0 baseline, three single-dimension variants, and a compound-scaled variant. These two visual examples are not a measurement of localization accuracy or object-detection recall.

All models start from EfficientNet-B0. The macaron and maze examples let us inspect where each model places its activation:

  • Single-dimension variants: in these two images, some hot regions cover only part of the object or shape.
  • Compound-scaled variant: the hot regions cover more class-relevant details. This is a visual observation; the classification results below are needed to assess the overall gain.

Additionally, the authors observed the performance of several scaling methods:

  1. Single-Dimension Scaling Methods:

    • Depth Scaling: Increasing the network depth, i.e., the number of layers.
    • Width Scaling: Increasing the network width, i.e., the number of channels per layer.
    • Resolution Scaling: Increasing the input image resolution.
  2. Compound Scaling Method:

    • Combining depth, width, and resolution growth, coordinated according to the proposed α\alpha, β\beta, γ\gamma parameters.

Figure 8 below compares scaling methods from the same EfficientNet-B0 baseline. Unpictured Table 7 lists configurations at about 1.8–1.9B FLOPs: depth, width and resolution alone reach 79.0/78.9/79.1% top-1, whereas compound scaling reaches 81.1%, a 2.0–2.2 percentage-point gap. The paper's “up to 2.5 points” refers to another comparison along the curves, not this Table 7 group.

Paper Figure 8: red is compound scaling; the other curves scale depth, width or resolution alone. The x-axis counts billions of FLOPs. Table 7's near-equal-cost configurations sit around 1.8–1.9B; not every plotted point has exactly the same compute cost.

Results on ImageNet

Paper Table 2: single-model, single-crop results are grouped by roughly similar accuracy. B7 and GPipe both show 84.3% top-1 with 66M and 557M parameters. GPipe has no FLOPs entry here, so its FLOPs ratio cannot be calculated from this table.

In this paper, the authors used the following specific training configurations:

  • Optimizer: RMSProp, decay and momentum both 0.9; BatchNorm momentum 0.99
  • Weight Decay: 1×10−51 \times 10^{-5}
  • Initial Learning Rate: 0.256, decaying by 0.97 every 2.4 epochs
  • Activation Function: Swish (the form used here is equivalent to SiLU)
  • Data Augmentation: AutoAugment
  • Regularization: stochastic depth with path survival probability 0.8

The authors increased dropout linearly from 0.2 for B0 to 0.5 for B7. They also set aside 25k training images for early stopping and reported results on the original validation set. Dropout and stochastic depth are distinct operations.

The savings vary by comparator. B7 reaches 84.3% top-1 with 66M parameters and 37B FLOPs; GPipe also reaches 84.3% but has 557M parameters, about 8.4 times as many. Table 2 omits GPipe's FLOPs.

For B3 versus ResNeXt-101, Table 2 reports 81.6/80.9% top-1 and 1.8/32B FLOPs, about an 18-fold operation-count difference in these paper results. It is not a latency ratio for every device.

Inference Speed

To validate latency, the authors also measured the inference latency of some representative CNNs on a single Intel Xeon E5-2690 CPU core with batch size 1.

The table below shows the average latency of 20 runs:

Paper Table 4: B1/ResNet-152 take 0.098/0.554 seconds, and B7/GPipe take 3.1/19.0 seconds. This table lists B1/B7 accuracy as 78.8/84.4%, different from Table 2's 79.1/84.3%; the values must not be combined as one evaluation.

Across 20 runs in that setup, B1 is about 5.7 times faster than ResNet-152 and B7 about 6.1 times faster than GPipe. These are one-core Xeon, batch-one results; GPU and mobile latency require new measurements. Table 4's accuracy values do not fully match Table 2, so they should not be presented as identical evaluation results.

Parameter-Accuracy Curve

Paper Figure 5: the left x-axis is parameter count, the right is FLOPs, and the red curves represent EfficientNet models. The wide imported image includes small inset tables that are part of Figure 5, not additional experiments.

Along the red curve from B0 to B7, accuracy grows with model scale. Compared with some plotted alternatives, EfficientNet reaches similar or higher top-1 accuracy at lower parameter count or FLOPs. The two x-axes differ, and latency requires separate measurement. In the right inset, NASNet-A is 80.7%, while Table 2 lists 82.7%; the paper does not explain this discrepancy here, so the main ImageNet comparison above follows Table 2.

Conclusion

The depth, width, and resolution of a model are crucial factors influencing the performance of convolutional neural networks.

The paper proposes compound scaling: coordinate changes to network depth, width and resolution to improve accuracy and efficiency. Table 7 and Figure 8 support this strategy when the scaling methods start from the same B0 baseline. B0 itself was found by NAS and the training setup was tuned, so not all of B7's advantage over older models can be assigned to the scaling equation.

Future work could refine the choice of compound-scaling parameters and explore whether the method helps other kinds of neural networks.