Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection
- Publication
- CVPR 2020Conference
- First submitted to arXiv
- arXiv:1912.02424
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{objectdetectionatss2019,
title = {{Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection}},
author = {Shifeng Zhang and Cheng Chi and Yongqiang Yao and Zhen Lei and Stan Z. Li},
year = {2019},
eprint = {1912.02424},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1912.02424}
}
In recent years, the development of object detection has split into two major factions: Anchor-based and Anchor-free.
Anchor-based Methods
- Two-stage detectors (e.g., Faster R-CNN) use RPN proposal regions + R-CNN refinement, maintaining a lead in accuracy over time through multiple iterations of regression.
- One-stage detectors (e.g., SSD, RetinaNet) lay anchors across multiple feature maps, directly performing classification and regression. This approach is extremely efficient and has gradually caught up with two-stage performance.
Anchor-free Methods
- Keypoint-based: Reformulate detection as a keypoint localization problem. For example, CornerNet determines boxes by "top-left + bottom-right" corners, while ExtremeNet directly locates four extreme points and the center.
- Center-based: Use the object’s center or region as the positive sample, then regress to the boundaries (e.g., YOLO, FCOS, FoveaBox).
Problem Definition
Though these two schools appear divergent, they are essentially answering the same question:
Which locations should be considered "positive samples," and which as "negative samples"?
Take RetinaNet (anchor-based) and FCOS (anchor-free, center-based) as examples. Their main differences lie in three areas:
- Sampling Unit: RetinaNet places multiple anchors at each location; FCOS places only a single point per location.
- Sample Definition: RetinaNet uses IoU thresholds to distinguish positives and negatives; FCOS relies on spatial and scale rules.
- Regression Origin: RetinaNet starts from anchor boxes; FCOS predicts outward from a single point.
However, through rigorous alignment experiments, the authors found:
What truly determines performance differences is not "box vs. point," but how positive and negative samples are defined.
If both methods use the same sample selection strategy during training, the final results are actually quite similar! In other words, the divide between anchor-based and anchor-free isn't about architecture—it's about assignment: the sample allocation strategy is the key!
Thus, a few crucial questions remain at the heart of object detection:
- Why rely on hand-crafted IoU thresholds or manually designed rules to divide positives and negatives?
- Is it possible for the model to automatically select the right samples based on each object's statistical properties?
And then, you can't help but ask: "What about anchors?"
Oh, they're already dead.
As witnesses to history, let's frame them and put them in a museum.
Problem Solving
The authors selected two representative methods for comparison: RetinaNet (anchor-based) and FCOS (anchor-free, center-based). On the surface, they have many structural differences, but to make a fair comparison, all "unnecessary distractions" must be eliminated one by one.
Removing Unfair Factors
To focus on the core difference, the authors modified RetinaNet so that each location has only one square anchor (referred to as RetinaNet #A=1), making its structure as close as possible to FCOS.
Even so, there was still a clear gap on COCO minival: RetinaNet (#A=1) achieved 32.5% AP, while FCOS had 37.1% AP.
At this point, the authors noticed that FCOS introduced many "general improvements":
- GroupNorm normalization
- GIoU loss
- Centerness branch
- Requiring positive anchors' centers to lie inside the GT box
- A learnable scalar for each feature pyramid layer
These techniques are not exclusive to anchor-free approaches and can also be applied to RetinaNet.
After adding these improvements, RetinaNet (#A=1) improved to 37.0% AP, nearly catching up with FCOS at 37.8% AP.
This alignment closes the original 5.3-AP gap to 0.8 AP, showing that many apparent differences came from implementation settings shared by both detectors.
Pinpointing the True Difference
After eliminating all "external advantages," only two differences remained:
- Sample Definition (classification)
- Regression Origin (regression)
Let's first look at the sample definition aspect.
Blue cells marked 1 are positives, blue question marks are spatial candidates awaiting the scale filter, and gray cells marked 0 are negatives.
- RetinaNet selects positives and negatives via IoU thresholds: each GT's best anchor and anchors with IoU are positives, anchors with IoU are negatives, and the intermediate range is ignored.
- FCOS uses spatial + scale conditions: points inside the box are candidate positives, then filtered by scale range at each layer, with the final survivors as positive samples.
Experimental results show:
- When RetinaNet (#A=1) switches to the "spatial + scale" rules of FCOS, AP increases from 37.0% → 37.8%.
- Conversely, when FCOS adopts the IoU threshold, AP drops from 37.8% → 36.9%.
Under these aligned conditions, changing the sample definition affects AP more than changing the box-versus-point regression origin.
Next is the regression origin aspect.
- RetinaNet predicts offsets from the anchor box.
- FCOS predicts four distances outward from a single point.
When both use the same sample assignment strategy in this controlled experiment, the regression-origin difference has little effect on AP:
- Under the IoU rule, box and point regression reach 37.0 and 36.9 AP; under spatial-and-scale assignment, both reach 37.8 AP.
Thus, the authors conclude:
- In this controlled comparison, sample definition affects AP more than starting regression from a box or a point.
Adaptive Training Sample Selection
Before introducing ATSS, let's define some notation:
-
: the set of all ground-truth (GT) boxes in the image.
-
: the set of all anchors; is the number of FPN layers, and anchors on each layer are .
-
: the main selection hyperparameter, the number of anchors nearest the GT center chosen per FPN level as candidates.
-
: for a GT , the collected candidate anchors ( in total).
-
: the IoU distribution between candidates and the GT.
-
, : the mean and standard deviation of .
-
: the adaptive IoU threshold for GT set by ATSS, defined as
To address the aforementioned issues, ATSS proposed a new idea:
Let each ground-truth object decide the boundary between positives and negatives for itself, instead of relying on manual settings.
The procedure is as follows:
- Select Candidates: For each GT , pick anchors per layer that are closest to its center, forming the candidate set .
- Compute Statistics: Calculate the IoU distribution among the candidates, and obtain and .
- Adaptive Threshold: Set .
- Select Positives: Any anchor with IoU ≥ and center inside the box is added to the positive set . If an anchor is chosen by multiple GTs, it is assigned to the GT with the highest IoU.
- All other anchors are negatives .
This design brings many benefits:
-
Center-based candidate selection: Whether anchor-based or anchor-free, candidates closer to the center are usually higher quality.
-
Using as the threshold:
- High → overall candidate quality is good → threshold should be high.
- Low → overall candidate quality is poor → threshold should be low to avoid missing GTs.
- High → only certain layers are suitable → raise the threshold, keeping only samples from those layers.
- Low → multiple layers are suitable → lower threshold, allowing more layers to participate.
-
Reducing differences in sample counts: In the authors' data, each GT has about positives, relatively stable across object size, aspect ratio, and position. This is an empirical observation.
-
Fewer main selection parameters: Results are similar for from about 7 to 17 in this experiment. Computing IoU with box-shaped anchors still requires their size; the authors also test size and aspect-ratio sensitivity.
To verify the effectiveness of ATSS on anchor-based detectors, the authors replaced the traditional sample assignment in RetinaNet (#A=1) with ATSS.
All evaluation metrics improved, as shown in the table below:
The authors attribute these improvements to each GT being able to adaptively select suitable positive samples based on its own statistical properties.
ATSS changes training-time sample assignment without changing the inference network, so it adds no inference overhead. Training still computes candidate IoUs and statistics.
Next, the authors applied ATSS to the anchor-free detector FCOS, leading to two versions:
-
Lite Version (Center Sampling)
- In original FCOS, all points inside the box are candidates, which introduces many low-quality positives.
- ATSS Lite: Select the nearest locations per layer as candidate positives, then retain the existing scale-range rule.
- This method has been integrated into official FCOS code as center sampling.
- Experimental results: AP improved 37.8 → 38.6 (as shown above).
- Downside: Still retains the scale range hyperparameter, not fully eliminated.
-
Full Version (Complete ATSS)
-
Temporarily treat FCOS points as square anchors with side length ( is the layer stride), then assign positives using the ATSS method.
-
During regression, prediction is still from the point to the four boundaries, same as original FCOS.
-
Experimental results:
AP +1.4 +1.7 +1.7 +0.6 +1.3 +2.7
-
Although Lite and Full use similar spatial candidates, they differ in how they select samples across scales. Full reaches 39.2 AP, above Lite's 38.6 AP, but Lite has 57.4 AP50 versus Full's 57.3; not every metric improves. The comparison supports the adaptive threshold over a fixed scale range for overall AP in this setup.
Discussion
To verify the final effectiveness of ATSS, the authors compared it with other mainstream detectors on MS COCO test-dev.
The experimental setup was as follows:
- Used multi-scale training (randomly scaling the short side of images to between and ).
- Doubled the total training steps to 180K, with learning rate reductions at 120K / 160K.
- All other settings were consistent with previous experiments.
Comparison with Existing Methods
-
With ResNet-101 as the backbone, ATSS achieved 43.6% AP, surpassing all other detectors using the same backbone:
- Cascade R-CNN: 42.8%
- C-Mask R-CNN: 42.0%
- RetinaNet: 39.1%
- RefineDet: 36.4%
-
With larger backbones:
- ResNeXt-32x8d-101 → 45.1%
- ResNeXt-64x4d-101 → 45.6%
-
ATSS is 0.1 percentage points below the table's SNIP (45.7 AP) row. SNIP uses a different backbone and multi-scale training/testing, so this compares reported scores under different conditions.
-
With DCN (Deformable Convolutional Networks):
- ResNet-101 → 46.3%
- ResNeXt-32x8d-101 → 47.7%
- ResNeXt-64x4d-101 → 47.7%
The strongest listed ATSS model with DCN reaches 47.7 AP in a single-model, single-scale test, above the other rows shown. With multi-scale testing it reaches 50.7 AP. These rows also change backbone, DCN, and test scale; their gains cannot all be attributed to sample assignment alone.
ATSS changes positive/negative assignment. Several backbone configurations with ATSS achieve strong AP and can combine with DCN and multi-scale testing, while cross-model rows alone do not quantify ATSS's isolated gain.
Discussion on Number of Anchors
Most experiments above were based on RetinaNet (#A=1), but traditional RetinaNet places 9 anchors per location (3 scales × 3 aspect ratios).
-
With the traditional IoU-based assignment:
- RetinaNet (#A=9) → 36.3% AP
- With general improvements (GroupNorm, GIoU loss, etc.) → 38.4% AP
- For comparison, RetinaNet (#A=1) achieves 37.0% AP.
- Conclusion: Under the traditional strategy, using more anchors does improve performance.
-
But after applying ATSS:
- RetinaNet (#A=9) rises from 38.4 to 39.2 AP, a gain of 0.8 percentage points.
- Results are nearly identical to RetinaNet (#A=1).
- Even if the number of scales or aspect ratios is reduced from 3 to 1, the performance hardly changes.
In this minival ablation, ATSS with one, three, or nine anchors per location all score 39.2–39.3 AP. More anchors show no visible AP advantage within these tested settings, leading the authors to question whether tiling multiple anchors is necessary here.
Conclusion
The greatest contribution of ATSS is refocusing the core issue of object detection on the definition of positives and negatives. Through a simple statistical rule, each GT can automatically determine its own sample boundary, eliminating reliance on fixed IoU thresholds or scale ranges.
ATSS also sharpened the long anchor-based versus anchor-free debate: in the authors' controlled tests, assignment mattered more than whether regression started from a box or point. It improved these detectors without changing their inference networks, though training still performs assignment work.
Related later work kept asking how to choose useful training examples:
- PAA and OTA / SimOTA explore probabilistic or optimization-based assignment; SimOTA appears in YOLOX.
- TOOD / TAL emphasize alignment between classification and localization during assignment.
- VFL and GFLv2 address how classification scores reflect localization quality. They are related scoring work, not direct ATSS assignment variants.
By 2025, many detector designs were exploring global optimization (OTA) and task alignment (TAL). Transformer-based architectures such as DETR and DINO use Hungarian matching instead. These are related ways of asking:
How can we ensure the model learns from the right samples?
We identify the problem, then solve the problem.
ATSS is undoubtedly a classic work worth reading.