Focal Loss for Dense Object Detection
- Publication
- ICCV 2017Conference
- First submitted to arXiv
- arXiv:1708.02002
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{objectdetectionretinanet2017,
title = {{Focal Loss for Dense Object Detection}},
author = {Tsung-Yi Lin and Priya Goyal and Ross Girshick and Kaiming He and Piotr Dollár},
year = {2017},
eprint = {1708.02002},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1708.02002}
}
Before this, the classic R-CNN methods dominated the object detection field. Although slow, they were highly accurate. Subsequent research aimed to address the speed issue with methods like SSD and YOLO.
These new methods were faster, all right, but they weren't accurate enough!
Can we have speed and accuracy at the same time?
Problem Definition
Why is it inaccurate?
The authors of this paper believe the root cause is the imbalance between positive and negative samples during training.
In single-stage object detection, the model evaluates numerous candidate locations and boxes, most of them background. The many easy negatives can collectively dominate the training loss and gradients even though each contributes little useful learning signal.
This issue isn't new, and some researchers have proposed Hard Negative Mining, which involves selecting difficult negative samples that are often misclassified as positives to train the model. These are called "hard negatives."
The authors take another route. Sampling hard negatives adds a selection step, while a loss that automatically reduces the weight of easy examples could address the imbalance directly.
They propose a new loss function called: Focal Loss.
Solution
Focal Loss
Cross-entropy loss (CE Loss) is a commonly used loss function for binary classification problems.
It is defined as follows:
where denotes the true class label and is the model's predicted probability for .
For simplicity, we define as follows:
Thus, cross-entropy loss can be rewritten as:
A notable characteristic of cross-entropy is that even well-classified examples with still incur some loss. Across a vast number of easy negatives, those small losses add up and can obscure the harder examples.
To address the class imbalance issue, a common approach is to introduce a weighting factor , assigning weights and to the positive and negative classes, respectively. In practice, can be set based on the inverse frequency of the classes or adjusted through cross-validation. Similar to the definition of , we define and rewrite the -balanced cross-entropy loss as:
This is the -balanced cross-entropy baseline used in the paper. Here is for the positive class and for the negative class.
Although can balance the importance of positive and negative samples, it cannot distinguish between easy and difficult samples.
Therefore, the authors propose reshaping the loss function to "down-weight" easy-to-classify samples, thereby focusing training on hard negatives.
This is done by adding a modulation factor to the cross-entropy loss, where is a tunable parameter.
The authors define Focal Loss as:
When a sample is misclassified and is small, the modulation factor is close to 1, and the loss is unaffected. However, as , the modulation factor approaches 0, reducing the loss for correctly classified samples.
The parameter smoothly adjusts the rate of down-weighting for easy samples. When , Focal Loss is equivalent to cross-entropy loss; as increases, the modulation factor's effect strengthens (the authors found to work best).
In practice, the authors use an -balanced variant of Focal Loss:
The authors found this form slightly more accurate than the non--balanced form.
The relationship between the classification score and the corresponding loss can be intuitively understood through the following chart:
In the chart, represents standard cross-entropy loss, while is the optimal value from the authors' experiments. The region from 0.6 to 1.0 corresponds to easy-to-classify samples, where the loss is effectively reduced after applying Focal Loss.
Model Architecture
To ensure the effect of Focal Loss is not overshadowed by model complexity, the authors designed a very simple model architecture called RetinaNet.
RetinaNet is a single-stage object detection model, with key design features including:
-
A simple backbone, such as ResNet50.
-
A straightforward neck, such as the Feature Pyramid Network (FPN).
-
Five feature-pyramid levels, through , with nine anchors at each location on each level. Separate subnetworks predict class scores and box offsets.
-
The anchors use three aspect ratios—1:2, 1:1, and 2:1—and three sizes at each level, scaled by , , and . That makes nine anchors, not nine aspect ratios. The following relative width:height examples show that scaling both dimensions leaves the ratio unchanged.
-
1:2 Aspect Ratio:
- 1:2
- 1:2
- 1:2
-
1:1 Aspect Ratio:
- 1:1
- 1:1
- 1:1
-
2:1 Aspect Ratio:
- 2:1
- 2:1
- 2:1
-
-
Each anchor receives a -dimensional class target and a four-dimensional box-regression target. The classification subnet outputs an independent sigmoid score for each class; box regression is class agnostic.
-
The RPN-style matching rule treats anchors with ground-truth IoU of at least as positives and those below as background. Anchors between the thresholds are ignored during training.
-
Each anchor yields one predicted box.
-
-
The object detection part is divided into two subnetworks, one for class prediction and one for bounding box regression, both composed of fully convolutional network architectures.
Training Parameters
- At inference, RetinaNet first filters candidates by class score , then takes at most the top 1,000 per FPN level before decoding. After combining levels, non-maximum suppression (NMS) at threshold produces the final detections.
- The authors found to work best for Focal Loss. They choose together with in their experiments; at this setting, works best.
- RetinaNet uses ResNet-50-FPN, ResNet-101-FPN, and ResNeXt-101-FPN as backbone networks.
- RetinaNet uses SGD as the optimizer, with a learning rate of 0.01, reduced to 1/10 at 60k and 80k iterations. The momentum is 0.9, and weight decay is 0.0001.
- Classification uses Focal Loss across all anchors, normalized by the number of matched positive anchors; box regression uses Smooth L1. The two losses are summed. The final classification-layer bias is initialized with a foreground prior of to stabilize early training.
Discussion
Experimental Results on the COCO Dataset
The ResNet-101-FPN RetinaNet row reaches 39.1 AP on COCO test-dev, 2.3 points above the table's strongest Faster R-CNN with TDM (Inception-ResNet-v2-TDM, 36.8 AP). Their backbones and training conditions differ; the table compares final model results.
Switching to ResNeXt-101-FPN raises the table result by 1.7 AP to 40.8 AP. Both RetinaNet rows use longer training and scale jitter than the basic ablation setup.
The 39.1-AP RetinaNet row also exceeds single-stage DSSD513 (33.2 AP) by 5.9 AP points. The figure below uses another speed/accuracy setup: RetinaNet-101-500 has 34.4 AP at 90 ms versus DSSD513's 33.2 AP at 156 ms, while RetinaNet-101-800 reaches 37.8 AP but takes 198 ms.
Ablation Studies
The authors conducted ablation studies on several key elements of RetinaNet, showing the following results:
-
Balanced Cross Entropy
In Table 1(a), the authors first try the -balanced cross-entropy loss described above. Raising from 0.5 to 0.75 changes AP from 30.2 to 31.1, a 0.9-point gain.
-
Focal Loss
Table 1(b) shows how the focusing parameter changes the modulation term. At , the loss reduces to cross-entropy; as grows, easy examples receive less weight. At , Focal Loss reaches 34.0 AP, 2.9 points above the best -balanced CE row of 31.1 AP in this ResNet-50 setup.
To ensure fair comparison, the authors found the optimal for each . The chart shows that with increasing , the chosen value is lower (since easy negatives are down-weighted, emphasizing positive samples less).
-
High-Density Two-Stage Systems May Be Unnecessary
As shown in table (c), the authors used ResNet-50 and scanned the number of scale and aspect ratio anchors used at each spatial position and pyramid level in FPN. They considered scenarios from one square anchor per position to 12 anchors per position, spanning 4 sub-octave scales ( for ) and 3 aspect ratios (0.5, 1, 2).
Even one square anchor achieves a surprising 30.3 AP. Three scales and three aspect ratios per position reach 34.0 AP, 3.7 points more; two scales and three aspect ratios actually reach 34.2 AP in the table.
All other experiments in this work use this setup.
Finally, increasing density beyond six to nine anchors per location shows no further gain in this test. Two-stage systems can classify boxes at arbitrary locations, sizes, and aspect ratios via region pooling; this ablation only shows that RetinaNet's anchor density is near saturation within the tested range.
-
Online Hard Example Mining (OHEM)
As shown in table (d), previous literature proposed improving two-stage detector training by constructing mini-batches containing high-loss samples. In OHEM, each sample is scored by its loss, then NMS is applied, and mini-batches are constructed with high-loss samples. NMS threshold and batch size are adjustable parameters. Like Focal Loss, OHEM emphasizes misclassified samples, but unlike FL, OHEM discards easy samples entirely.
In the same ResNet-101-FPN ablation setup, the best OHEM configuration (no forced 1:3 ratio, batch size 128, NMS 0.5) reaches 32.8 AP. Focal Loss reaches 36.0 AP, 3.2 points higher. The authors also tried other OHEM settings and variants without improving that result.
-
Larger Backbone
In Table 1(e), larger backbones can improve accuracy but slow inference. Increasing input scale, defined by the image's shorter side, has a similar effect. This part of Table 1 uses COCO test-dev, whereas the earlier ablations generally use minival, so their numbers should not be treated as one fixed-condition sweep. Improving the architecture further for faster inference lies beyond this paper's scope.
Conclusion
RetinaNet effectively addresses class imbalance by introducing Focal Loss, allowing the model to focus on hard negatives and significantly improving detection accuracy. Experimental results show RetinaNet surpasses various existing one-stage and two-stage detection methods on the COCO dataset, demonstrating its practical potential.
The later YOLOv3 report also tried Focal Loss, but its own experiment reported a roughly two-point mAP decline. That is a result for a different detector and setup; whether the architecture explains the contrast remains uncertain.
There is still much work to be done in object detection research, and we look forward to future advancements.