Precise Detection in Densely Packed Scenes
- Publication
- CVPR 2019Conference
- First submitted to arXiv
- arXiv:1904.00853
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{retailproductsku110k2019,
title = {{Precise Detection in Densely Packed Scenes}},
author = {Eran Goldman and Roei Herzig and Aviv Eisenschtat and Oria Ratzon and Itsik Levi and Jacob Goldberger and Tal Hassner},
year = {2019},
eprint = {1904.00853},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1904.00853}
}
This is also an earlier piece of literature.
However, the benchmark established in this paper has become a recurring benchmark in subsequent work on dense retail-product detection.
Problem Definition
Modern object detection techniques have demonstrated very high accuracy and speed in natural scenes. However, mainstream methods struggle when faced with densely arranged, visually similar, and closely adjacent objects.
Such "high-density scenes" are actually very common, for example: retail shelf displays, urban traffic surveillance footage, or indoor warehouse and stacking environments.
These scenes typically have several notable characteristics:
- A very large number of objects, arranged tightly and often occluding each other.
- Highly similar appearances, making boundary distinction difficult.
Traditional object detection methods, such as Faster R-CNN, YOLO, RetinaNet, etc., perform well in natural scenes but mostly rely on post-processing steps like Non-Maximum Suppression (NMS) to eliminate overlapping boxes. This hard suppression strategy shows clear bottlenecks in dense scenes. For example, a single real object might be split into multiple small boxes, or several adjacent objects might be mistakenly detected as one.
Although previous research has proposed various alternatives such as Mean-Shift, Agglomerative clustering, or learnable NMS layers (e.g., GossipNet), most methods are either computationally expensive or fail to truly replace traditional NMS, especially in high-density, visually similar object scenarios.
Furthermore, current mainstream datasets (e.g., COCO, VOC, Open Images) mainly contain natural scenes and lack good standards for these "dense, similar, overlapping" special scenes. Even datasets emphasizing crowd or vehicle congestion, like CARPK and PUCPR+, fall far short in item density and classification difficulty compared to real retail environments.
So, when facing these "crowded, look-alike, and numerous" scenes, how can deep learning models accurately distinguish each individual object while maintaining efficiency and stability?
Can we find a way to let the model “know how precise its bounding boxes are”?
The authors of this paper propose: Let’s start with Soft-IoU.
Solution
The main baseline compared in this paper is RetinaNet, because it already performs quite well in dense scenes but still has room for improvement. The authors extend the standard detection architecture with two additions:
- Introduce a Soft-IoU prediction head: allowing the model to estimate “how much this box overlaps with the ground truth”;
- Propose an EM-Merger inference unit: converting multiple detection boxes into Gaussian distributions, then resolving overlaps via probabilistic clustering.
Together, these two designs reduce overlapping detections in dense scenes.
Basic Architecture
The authors use ResNet-50 as the backbone and build a Feature Pyramid Network (FPN) with three upsampling layers to generate multi-scale feature maps.
At each feature level, this RetinaNet-style single-stage detector has three convolutional output heads; these are not a separate two-stage RPN:
- Detection head: regress bounding box center and size:
- Classification head: predict objectness confidence
- Soft-IoU head: predict the overlap degree between the box and the true object
To avoid introducing excessive noise when training Soft-IoU, candidate boxes with classification scores below 0.1 are filtered out.
Soft-IoU
In non-crowded scenes, objectness score + NMS is usually sufficient. But in dense scenes, multiple high-confidence boxes may correspond to different objects or be false detections caused by occlusion, so a more refined “overlap quality metric” is needed.
Soft-IoU is designed for this purpose. It predicts the IoU score between each box and its ground truth box. For each detection box , the closest ground truth box is found, then:
If there is no intersection, IoU is set to 0.
This value serves as ground truth to train the Soft-IoU prediction head with a binary cross-entropy loss:
The overall loss is the sum of three terms:
where the first two are standard classification and regression losses.
A special point is that objectness predicts “whether there is an object,” while Soft-IoU predicts “how precise this box is.” Thus, a box partially covering an object may have a high objectness score but a lower Soft-IoU. These two signals complement each other.
EM-Merger
Through Soft-IoU, each detection box has two confidence values:
- : presence of an object
- : accuracy of the box
The problem is these boxes may overlap and cluster together.
How do we precisely find each object from these overlapping candidates?
The authors’ solution is to treat each detection box as a 2D Gaussian distribution:
where:
- Mean : box center;
- Covariance : reflecting box size.
These Gaussians form a mixture model (MoG):
Here the covariance follows the coordinate order (x, y): width corresponds to x and height to y. The paper’s displayed height/width ordering uses a different convention; coordinate order must be kept consistent in implementation. The mixture is a spatial density constructed from boxes, not a calibrated probability that a given pixel is an object center.
Next, the MoG is reduced to a smaller number of high-confidence Gaussians representing the true object detections. The goal is to approximate the original distribution with by minimizing:
where KL is the Kullback–Leibler divergence.
The authors initialize the clusters with hierarchical agglomerative clustering, then approximately solve this minimization with EM-like steps:
-
E-step: assign each detection box to the nearest cluster (by KL divergence)
-
M-step: re-estimate each cluster’s parameters
The iteration stops on convergence or at the configured cap of 10 iterations; convergence within 10 iterations is not guaranteed. The initial cluster count is bounded by the image area divided by the area of a mean-sized box. After clustering, lower-confidence Gaussians overlapping others beyond a threshold are suppressed, leaving detections.
Finally, for each cluster center, an ellipse with twice the standard deviation is drawn. The original detection boxes inside this ellipse are collected, and the median width and height are taken as the final bounding box.
This is the overall inference pipeline: Soft-IoU as quality prediction → MoG expressing a spatial distribution → EM clustering and overlap suppression → fewer neighboring objects discarded and fewer duplicate detections.
SKU-110K Dataset and Benchmark Construction
The lack of breakthroughs in dense detection is partly due to the absence of high-quality datasets specifically designed for this task.
To address this, the authors created a brand-new dataset, SKU-110K, specifically targeting retail shelf environments. From design motivation to annotation quality, it was planned with the characteristics required for high-density scenes.
The dataset images were collected from thousands of physical stores across the United States, Europe, and East Asia, taken by dozens of photographers using personal smartphones to ensure the images reflect natural, unarranged real-world shelf displays.
- Each original image has a resolution of at least 5MP, later compressed to 1MP JPEG;
- No restrictions were placed on capture devices or angles, preserving natural lighting, blur, occlusions, and other real-world disturbances;
- Annotation was performed by skilled full-time personnel rather than crowdsourcing, further improving annotation accuracy and consistency;
- All images and annotations were manually reviewed to filter out obvious mislabels and incorrect boxes.
Dataset Statistics and Splits
SKU-110K contains three subsets, ensuring that shelves from the same store do not appear in multiple subsets:
| Dataset | Images | Boxes | Ratio |
|---|---|---|---|
| Training | 8,233 | 1,210,431 | 70% |
| Validation | 588 | 90,968 | 5% |
| Testing | 2,941 | 432,312 | 25% |
The split counts imply about 148 annotated objects per image overall, illustrating the dataset’s density. A blanket claim of one to three orders of magnitude above other datasets is not justified by these counts alone.
Evaluation Metrics and Criteria
The main evaluation criteria follow COCO standards, including:
- AP@[.50:.95]: primary challenge metric;
- [email protected]: strict localization accuracy;
- AR@300: average recall limited to 300 predictions;
- [email protected] (IoU=0.75): precision at 50% recall sample point.
Additionally, to balance detection and counting performance, the authors introduced error metrics:
-
MAE (Mean Absolute Error)
-
RMSE (Root Mean Squared Error)
where is the model’s predicted object count per image, and is the ground truth count.
The photographs contain many product appearances, but the benchmark uses one foreground object class for localization and counting. It does not evaluate recognition of individual SKU identities or fine-grained product classification.
Discussion
Inference Speed Analysis
To verify model deployability, the authors measured average inference performance on identical hardware (Intel i7-5930K + GTX Titan X), using metrics:
- FPS (Frames Per Second): number of images processed per second;
- DPS (Detections Per Second): number of objects detected per second.
Results:
- Base Detector (RetinaNet isomorphic): baseline inference speed;
- Adding Soft-IoU Layer: the same 0.5 FPS as the base detector in this table;
- EM-Merger: drops to 0.23 FPS, largely because of frequent exchanges between GPU and CPU/RAM.
Although EM-Merger is currently slower, its accuracy gain justifies deployment optimization. The authors note that implementing it on GPU could greatly improve speed.
Detection and Counting Experiments
Several comparisons and ablations were designed:
- Monkey: randomly throws boxes while given the object count and box-size statistics as a comparison;
- Base & NMS: base architecture + traditional NMS;
- Soft-IoU & NMS: NMS using Soft-IoU scores;
- Base & EM-Merger: retaining original objectness, only EM clustering;
- Full System (this paper’s method): Soft-IoU + EM-Merger.
Detection results:
- Monkey is almost ineffective, proving density alone is insufficient;
- Faster R-CNN / YOLO9000opt perform poorly under this dense benchmark configuration; 50 objects is not a hard architectural limit;
- RetinaNet performs relatively stably and slightly better than the authors’ Base.
The full method exceeds the baselines on all the table's detection metrics ([email protected]–.95, [email protected], AR@300, and P@R=.5). Merely adding Soft-IoU while keeping NMS offers only small gains; its AR@300 is even slightly below Base & NMS.
For counting, the predicted number is compared with the true count . RetinaNet records 16.584 MAE / 30.702 RMSE, against 14.522 / 23.992 for the full method. The authors attribute part of the gap to greedy NMS. But Base & EM-Merger has 283.971 RMSE in Table 3, showing that EM clustering alone is not enough.
This indicates Soft-IoU provides a better candidate evaluation criterion, while EM-Merger effectively separates crowded areas. Together, they substantially improve counting stability.
Below are sample SKU-110K test images comparing the method with RetinaNet. In these examples, the method reduces overlapping and mislocalized boxes, producing more one-box-per-object results.
Generalization Experiments on CARPK and PUCPR+
To verify generalization, the authors applied their method to two vehicle counting datasets: CARPK and PUCPR+.
Results show that although this method was not specifically designed for vehicle counting, it still achieves excellent performance on both datasets, outperforming other methods in MAE and RMSE.
Conclusion
This study proposes the Soft-IoU layer and EM-Merger module as concrete engineering solutions to overlapping box issues in dense scenes. The former complements the blind spot of objectness which cannot measure boundary precision, while the latter replaces traditional NMS with probabilistic clustering to enhance localization clarity and deduplication.
On SKU-110K, the method outperforms contemporaneous mainstream architectures on both detection and counting metrics, confirming its effectiveness. However, from today’s perspective, main limitations include:
- Single-foreground-class evaluation: additional category supervision and evaluation would be needed for SKU recognition;
- EM-Merger is challenging to deploy in real-time systems, inference speed still needs optimization;
- Lacks integration of semantic information, still prone to confusion in fine-grained appearance variations.
Although published years ago, this method’s contribution to dense object detection remains valuable. Especially SKU-110K’s extreme crowded scenes remain common in many domains today (e.g., self-checkout, smart warehousing, traffic monitoring), serving as a good benchmark for evaluating model generalization.