You Only Look Once: Unified, Real-Time Object Detection
- Publication
- CVPR 2016Conference
- First submitted to arXiv
- arXiv:1506.02640
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{objectdetectionyolov12015,
title = {{You Only Look Once: Unified, Real-Time Object Detection}},
author = {Joseph Redmon and Santosh Divvala and Ross Girshick and Ali Farhadi},
year = {2015},
eprint = {1506.02640},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1506.02640}
}
Object detection systems usually divide the work into two main steps: region proposals and classification.
-
Region proposals:
This step produces candidate boxes that might contain objects.
R-CNN, for example, uses Selective Search to generate thousands of candidate regions. Faster R-CNN instead generates them with a region proposal network (RPN), making proposal generation trainable and much faster. It still processes proposed regions in a further detection step; it does not rerun the entire image network separately for every candidate.
-
Classification:
After proposals are made, a classifier examines candidate regions to decide which object class each contains. R-CNN extracts convolutional features separately for each region and then uses an SVM. Later detectors share image features to avoid that repeated computation, but their staged proposal and classification process still takes time.
Problem Definition
The question is: can we combine proposals and classification so that the system finds objects after looking at the image just once?
YOLO reframes object detection as a single regression problem, predicting bounding box coordinates and class probabilities directly from the full image. The design has several attractions:
- One network evaluation: A single forward pass produces candidate boxes and class probabilities, simplifying the detection pipeline. Thresholding and non-maximum suppression still follow the network.
- Global context: The model sees the entire image during training and inference, so its predictions can use surrounding information. The paper's error analysis finds fewer background mistakes.
- Real-time speed: The resulting speed and accuracy trade-off makes detection feasible at real-time frame rates under the paper's hardware and dataset conditions.
The faster pipeline also asks the network itself to learn localization. That trade-off will matter when we examine the errors.
Solution
Model Architecture
To combine region proposals and classification, the authors use a convolutional backbone followed by two fully connected layers. The convolutional layers extract image features; the fully connected layers combine them and project them to the detection output.
The YOLOv1 backbone has 24 convolutional layers, inspired by GoogLeNet and using alternating reduction and convolutions. The authors use the Darknet framework to train it. Darknet-19 is a later YOLOv2 architecture, not the network in this figure. Fast YOLO has a smaller, 9-convolutional-layer version.
For ImageNet classification pretraining, the authors use the first 20 convolutional layers with a pooling and classification head at . For detection they add four convolutional layers and two fully connected layers, then increase the input to .
Look at the block on the right of Figure 3. It is the output tensor, not a input image: the first two dimensions correspond to a grid over the original image. Figure 2 makes the idea easier to see:
Each grid cell has a few jobs:
- Predict two bounding boxes, each with and confidence. The center coordinates are relative to that cell's bounds; width and height are relative to the whole image. Confidence combines object presence with the box's predicted IoU.
- Predict one set of class probabilities conditional on an object being in the cell. PASCAL VOC has 20 classes, and both boxes share the same 20 probabilities.
The paper sets three values:
- : the number of cells along each image side, here 7.
- : the number of box predictors per cell, here 2.
- : the number of classes, here 20.
The resulting output is . If an object's center falls in a cell, that cell is responsible for detecting it. Of its two box predictors, the one with the highest current IoU against the ground-truth box takes responsibility for coordinate regression.
Each box confidence is . At inference, multiplying it by a cell's conditional class probability yields a class-specific score for that box.
Loss Function
The predicted boxes can overlap. During training, the model first assigns responsibility to a box predictor; during inference, it later filters duplicate detections. These are different steps.
The authors train with a squared-error loss made of five parts. Let index grid cells and their box predictors. is one when predictor is responsible for a ground-truth object; means the cell contains an object. Hats denote targets, is box confidence, and is the cell's class probability. With and , Equation 3 can be written together as:
Let's take those five pieces one by one.
-
Coordinate loss:
The responsible predictor pays squared error for the center . The weight increases the importance of localization, and keeps other predictors out of this coordinate term.
-
Size loss:
The same responsible predictor pays for width and height, but the error is measured after taking square roots. This reduces how much large boxes dominate the absolute error. It does not make small and large boxes equally weighted, nor does it directly optimize IoU.
-
Responsible-box confidence loss:
This compares the responsible box's confidence with its target. When a cell contains an object, that target is the predicted box's IoU with the ground truth. It is a confidence target, not a class probability.
-
No-object confidence loss:
This pushes confidence toward zero for box predictors that are not responsible, including boxes in empty cells and an unassigned predictor in an occupied cell. The weight prevents the many background boxes from overwhelming the object signal.
-
Classification loss:
The one conditional class distribution is compared with class targets only when the cell contains an object. The term is per cell, rather than separately for each of the two boxes.
So predicting two boxes does not make both of them pay coordinate loss for the same object. The unassigned predictor can still receive a no-object confidence penalty.
Inference Scenario
Each of the cells emits boxes, giving candidate boxes in this setting. The network first combines each box confidence with the cell's conditional class probabilities to obtain class-specific scores. It then thresholds detections and applies non-maximum suppression (NMS) to overlapping boxes of the same class.
NMS is a postprocessing operation after the single network forward pass. The paper reports an approximately 2–3% mAP gain from using it. Its basic steps are:
- Start with boxes of one class and their class-specific scores.
- Sort them by score.
- Keep the highest-scoring remaining box.
- Suppress boxes whose IoU with it exceeds the chosen threshold.
- Repeat selection and suppression until no boxes remain.
Let's work through four boxes. Their coordinates are , and the 0.5 threshold here is only an illustration, not a YOLOv1 hyperparameter specified by the paper.
| Box | Class score | Top-left | Bottom-right |
|---|---|---|---|
| A | 0.90 | ||
| B | 0.75 | ||
| C | 0.60 | ||
| D | 0.50 |
Step 1: Sort
By score, the order is A (0.90), B (0.75), C (0.60), then D (0.50).
Steps 2 and 3: Select and suppress
Keep A first. Then compare A with the other boxes. Their coordinates give and , both above the illustrative 0.5 threshold, so suppress B and C. D does not overlap A and remains.
Repeat the selection and suppression
Of the remaining boxes, choose D. Nothing else remains to compare, so the result is A and D. With more candidates, the same procedure would continue.
The example shows how NMS removes duplicate detections while retaining the highest-scoring box in an overlapping group. Its actual effect depends on class scores, box geometry, and the threshold.
Discussion
Inference Speed
Table 1 compares speed and mAP on PASCAL VOC 2007. The 30Hz and 100Hz DPM rows are GPU versions trained on VOC 2007; Fast YOLO is trained on VOC 2007+2012. Fast YOLO reaches 155 FPS and 52.7 mAP, more than twice the mAP of either listed real-time DPM version. Their training data differ, so that gap is not an architecture-only comparison.
Standard YOLO reaches 45 FPS and 63.4 mAP, still real-time in the paper's Titan X setup. Replacing its network with VGG-16 raises mAP to 66.4 but lowers speed to 21 FPS: this paper's focus is the faster model.
The fastest DPM row under “Less Than Real-Time” runs at 15 FPS; R-CNN Minus R and Fast R-CNN are also below real-time. Faster R-CNN with VGG-16 reports 73.2 mAP at 7 FPS, roughly six times slower than YOLO under the table's measurement. Its smaller ZF version reports 62.1 mAP at 18 FPS, about 2.5 times slower. Those are specific model and hardware comparisons, not timeless speed rankings.
Comparison with Other Methods
To see where YOLO differs from other detectors, the authors analyze publicly available VOC 2007 detections. For each of the 20 classes they take the top predictions, where is the number of true objects of that class, and classify each selected prediction as:
- Correct: correct class and IoU .
- Localization: correct class but .
- Similar: a similar class and IoU .
- Other: a wrong, non-similar class and IoU .
- Background: IoU with every true object.
Figure 4 averages these categories across classes. They are shares of selected top-ranked detections, not false-positive rates measured over every image region.
YOLO has trouble locating objects precisely. Localization errors make up 19.0% of its selected detections, more than its other error categories combined, versus 8.6% for Fast R-CNN. Fast R-CNN has 13.6% background errors against YOLO's 4.75%, about 2.9 times as large. The authors relate the background difference to YOLO seeing the whole image; the chart itself establishes an error-distribution difference.
YOLO's speed therefore comes with more localization errors, while Fast R-CNN localizes more accurately among these detections but mistakes background for objects more often. The methods trade different kinds of error rather than one simply winning every metric.
Limitations
YOLO imposes a strong spatial constraint: each grid cell predicts only two boxes and a single class distribution. That limits the number of nearby objects it can distinguish. Think of a flock of small birds whose centers crowd into the same cell.
The model also learns box shapes from training examples, so unusual aspect ratios and arrangements can be difficult to generalize to. Multiple downsampling stages leave relatively coarse features for localization, especially of small objects.
Finally, squared error is not the same as final detection performance. The same coordinate error can hardly matter for a large box but greatly reduce IoU for a small one. Square-root width and height terms only partly reduce the scale imbalance; they do not remove YOLO's difficulty with precise localization.
Conclusion
YOLOv1 treats detection as a single regression problem, predicting boxes and class probabilities directly from image pixels with one network. The base model reaches 45 FPS in the paper's setup, and Fast YOLO reaches 155 FPS.
The unified design makes real-time detection practical while preserving useful accuracy, but small and crowded objects remain difficult to localize. The paper's analysis finds fewer background errors and more localization errors than Fast R-CNN among selected VOC detections. Those strengths and limits helped establish a foundation for later real-time detectors.
“You Only Look Once” is a good name for the network evaluation. Score filtering and NMS still complete the final detections afterward.