EfficientDet: Scalable and Efficient Object Detection
- Publication
- CVPR 2020Conference
- First submitted to arXiv
- arXiv:1911.09070
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{featurefusionbifpn2019,
title = {{EfficientDet: Scalable and Efficient Object Detection}},
author = {Mingxing Tan and Ruoming Pang and Quoc V. Le},
year = {2019},
eprint = {1911.09070},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1911.09070}
}
About half a year earlier, Google had introduced NAS-FPN and let a search algorithm discover the feature-fusion paths. Now another Google team goes back to drawing the paths by hand. Wait—did they give up on search already?
The paper gives a practical reason: NAS-FPN costs a lot to search and produces an irregular structure that is hard to explain or adjust. So the authors leave the searched topology behind and draw a new fusion module from FPN and PANet: BiFPN.
Defining the Problem
Since its inception, the pioneering work of FPN has been widely applied to various object detection tasks. Numerous variants and improvements have followed, and we can see a rough development trajectory in the image below:
Starting from the original FPN on the far left, feature fusion occurs across different resolution feature maps, with the deepest feature map being progressively upsampled and added back to shallower feature maps.
PANet later proposed that simply adding features back along one path was insufficient, so they introduced a second path: another round of downsampling and addition. The purpose of this, as we’ve discussed before, is to establish a shortcut for feature fusion from the shallowest to the deepest layers, allowing the model to better utilize different levels of features.
NAS-FPN then used automated search to select a multi-layer fusion structure from candidate designs. The search is costly, and its resulting connections are harder to interpret and adjust.
In this paper, the authors from Google decided not to use the NAS approach but instead revisited the strengths of PANet, focusing on two key perspectives:
- Feature fusion shouldn’t be just a simple addition; it needs to consider the importance of features and perform weighted addition.
- The number of FPN layers should scale with the model size.
Solving the Problem
Model Architecture
EfficientDet follows the single-stage direction of detectors such as RetinaNet, using an EfficientNet backbone, repeated BiFPN layers, and class/box prediction heads whose weights are shared across scales. The full model is more than an FPN replacement; this note focuses on the fusion network.
In the image above, the backbone on the far left is not the focus of this paper, nor is the head on the far right. We should focus solely on the BiFPN in the middle.
This design starts with PANet and makes a few changes. Let's sketch them out:
-
Cross-scale connections:
The design retains bidirectional cross-scale paths and adds shortcuts between input and output nodes at the same scale. Learnable weights control each input’s contribution at fusion nodes.
-
Removing low-contribution nodes:
The authors remove nodes with only one input edge because they do not fuse multiple sources. This is an efficiency choice, not a claim that every single-input operation is useless.
These two pictures annotate the paper's PANet diagram to explain design steps; they are not separate original-paper BiFPN results. After adding shortcuts and removing single-input nodes, return to Figure 2(d) for the final BiFPN topology.
The authors also emphasized an important point: they view this structure as a “module.” As such, it can be stacked in layers, allowing for more feature fusion layers as the model size increases.
Model Scaling
The compound-scaling approach extends EfficientNet by jointly adjusting the backbone, BiFPN, prediction heads and input resolution, rather than enlarging only one component.
If the backbone scales, the fusion network and prediction heads should be configured with it. Table 1 goes from D0 at 512-pixel input, 64 BiFPN channels and 3 layers to D6 at 1280-pixel input, 384 channels and 8 layers. D7 and D7x retain the same BiFPN width and depth; D7x uses the larger B7 backbone and an extra feature level. The layer count therefore does not increase at every model step.
Weighted Average
Finally, the authors discussed how to design the weights for the weighted feature maps.
Three design methods were proposed in the paper:
- Unbounded fusion: Learn unconstrained weights directly. This can destabilize training, motivating normalization.
- Softmax fusion: Apply softmax to the weights, ensuring they sum to 1.
- Fast normalized fusion: Apply ReLU so , then use with . The epsilon makes the weight sum approximately, not exactly, 1 and prevents division by zero when all weights are zero.
The blue and red dashed curves above trace learned input weights at three BiFPN nodes during training. They show similar weight behavior, not accuracy or runtime. Those comparisons are in Table 6: softmax versus fast fusion produces 33.96/33.85, 43.78/43.77, and 48.79/48.74 AP across three models, with GPU speed ratios of 1.28, 1.26, and 1.31. The authors therefore choose the faster fusion with similar accuracy.
Discussion
Performance on COCO
The authors conducted experiments on the COCO 2017 detection dataset, using 118,000 training images. Results including D7x below come from the revised paper and retain its historical settings.
During training, they used SGD with momentum 0.9 and weight decay 4e-5. The learning rate rose linearly from 0 to 0.16 during the first epoch, then followed a cosine decay schedule. Synchronized batch normalization followed each convolution, with decay 0.99 and epsilon 1e-3.
The model employed SiLU (Swish-1) as the activation function and applied an exponential moving average with a decay factor of 0.9998 to improve stability. During training, the focal loss function was used, with α set to 0.25 and γ to 1.5. Data augmentation strategies included horizontal flipping and scale jittering, with image sizes randomly adjusted to 0.1 to 2.0 times the original size before cropping. Furthermore, soft non-maximum suppression (Soft NMS) was used during evaluation.
For training resources, EfficientDet-D0 to D6 models were trained for 300 epochs on 32 TPUv3 cores, while EfficientDet-D7 and D7x models were trained for 600 epochs on 128 TPUv3 cores, with a total batch size of 128.
Table 2 groups models by roughly similar accuracy. Across its different pairings, EfficientDet can have 4–9 times fewer parameters and 13–42 times fewer FLOPs (multiply-adds); those endpoints do not all describe one model. D0 versus YOLOv3 is 34.6 versus 33.0 test-dev AP and 2.5B versus 71B FLOPs, about 28 times fewer operations. The up-to-eightfold parameter and twenty-onefold FLOP comparisons with RetinaNet and Mask R-CNN also refer to different roughly matched pairs.
At the high-accuracy end, D7x scores 55.1 AP on COCO test-dev and 54.4 AP on val. The table lists 50.7 val AP for AmoebaNet+NAS-FPN+AA, a 3.7-point gap on the same split, or roughly four points; subtracting its val score directly from D7x's test-dev score would mix splits. D7x uses 410B FLOPs versus 3045B for that comparison model, about one seventh as many operations, not a measured sevenfold speedup. The paper also reports latency on Titan V FP32, V100 FP16, and a single-thread CPU; its V100 figures include preprocessing and NMS. Hardware, precision, and measurement boundaries matter here.
Conclusion
Under the same RetinaNet training setup, paper Table 4 moves from ResNet50+FPN at 37.0 AP to EfficientNet-B3+FPN at 40.3 AP, then EfficientNet-B3+BiFPN at 44.4 AP. Both changes contribute; the entire EfficientDet-family gain cannot be credited to BiFPN alone. In Table 5, with backbone, prediction heads, and training held fixed, weighted BiFPN reaches 44.39 AP versus 43.94 AP without weights. In the tested settings, its bidirectional paths, selective weighting, and lighter fusion network improve the accuracy-cost tradeoff.
Simple and effective, BiFPN is a great companion for implementing models. We highly recommend giving it a try.