MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors
- Publication
- CVPR 2023Conference
- First submitted to arXiv
- arXiv:2211.09791
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingmotrv22022,
title = {{MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors}},
author = {Yuang Zhang and Tiancai Wang and Xiangyu Zhang},
year = {2022},
eprint = {2211.09791},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2211.09791}
}
MOTR made multi-object tracking fully end-to-end: track queries are passed down frame by frame, new objects are handed to detect queries, and no matching or post-processing is needed.
On DanceTrack, MOTR's association ability was the best of its time, with an AssA of 40.2, higher than methods that match by motion, like ByteTrack and OC-SORT. But its detection wasn't good. Its DetA on DanceTrack was only 73.5, well below OC-SORT's 80.3; on MOT17, its MOTA of 73.4 was nearly 7 points behind ByteTrack's 80.3.
Yuang Zhang of Shanghai Jiao Tong University and Tiancai Wang and Xiangyu Zhang of MEGVII, members of the original MOTR team, think the problem is that the same decoder has to learn two things at once. Detection and association pull against each other inside MOTR, and detection ends up poorly learned.
So why not bring in an off-the-shelf, very strong detector to help?
Defining the problem
The most direct way to hook an external detector up to MOTR is to copy TransTrack: have the track queries predict boxes, then match them to the detector's boxes using IoU.
The authors tried it. Detection improved only slightly, and it went back down the old road of "predict first, then match," losing MOTR's end-to-end nature.
What the authors want is something like tracking-by-detection, taking detection results as input so that MOTR only has to concentrate on learning association. How to "feed" detection results into a query-centered Transformer is the problem this paper sets out to solve.
Solving the problem

MOTRv2 consists of an off-the-shelf YOLOX detector and a version of MOTR rewritten with anchor-based queries.
Anchor-based queries
The key step comes from the idea behind detectors like DAB-DETR. A query isn't just an abstract embedding; it also carries an explicit anchor box (center, width, height). The decoder only has to predict an offset relative to the anchor box, which makes detection much easier.
The authors rewrite both MOTR's detect queries and its track queries in this anchor-based form. The original learnable positional embeddings are replaced by sine-cosine encodings of the anchor box coordinates.
That way, an external detector's boxes can be used directly as anchor boxes, and the detection prior gets plugged into MOTR.
Proposal queries

In each frame, YOLOX produces candidate boxes, each with a center , width and height and a confidence score . To miss as few objects as possible, any box with a confidence score above 0.05 is kept.
Proposal queries are produced in three steps:
- A learnable shared query of size is copied into .
- Each candidate box's confidence score is encoded with sine-cosine encoding into a -dimensional vector.
- The two are added together to give this frame's proposal queries; the candidate boxes themselves serve as their anchor boxes.
All proposal queries share the same embedding, and they differ from one another only through their anchor box positions and confidence scores. In addition, 10 learnable anchor boxes are kept to pick up objects YOLOX misses.
The authors also compared whether to encode the confidence score at all. Without scores, validation HOTA is 63.0; with a linear projection, 63.4; with sine-cosine encoding, 63.7. The confidence score is useful information for MOTR.
Passing boxes along
The first frame has only new objects. The proposal queries go through the decoder, which outputs offsets relative to the YOLOX boxes; added back, these give the predicted boxes , and the updated queries become track queries .
In every frame after that, the previous frame's track queries are concatenated with this frame's proposal queries ; the previous frame's predicted boxes are likewise concatenated with this frame's YOLOX boxes , serving as the anchor boxes for all the queries.
That settles the division of labor. Proposal queries are responsible only for objects that are new or were previously missed, and track queries are responsible for relocating objects already being tracked. Coordination between the two is left to the decoder's self-attention. Proposal queries need to look at the track queries to avoid re-detecting someone who is already being tracked; track queries can also consult the YOLOX boxes to get their positions more accurate.

The authors plot the self-attention map. Attention between the proposal query and the track query of the same person is clearly higher, so the two really are exchanging information.
The original MOTR passes along only the center point as a reference point, while MOTRv2 passes along the whole box. The authors compared the two: measured by AssA on the DanceTrack validation set, passing only the center point gives 49.2, and passing the whole box raises it to 53.1. The width and height of the previous frame's box help association a lot.
Training
During training, each short clip has 5 frames, with the interval between frames chosen at random from 1 to 10.
The original MOTR artificially inserts fake track queries or randomly drops real ones to simulate false positives and missed detections. MOTRv2 no longer does this; it simply passes on track queries whose confidence score is above 0.5. The false positives and missed detections that arise naturally during training are the most realistic augmentation there is.
On DanceTrack and MOT17, YOLOX itself isn't trained further; the weights officially released by DanceTrack and ByteTrack are used directly, and the candidate boxes for all images are computed before training.
Two more training tricks:
- Mixing in CrowdHuman. As in MOTR, single CrowdHuman images are turned into fake 5-frame videos and trained together with DanceTrack.
- Query denoising. The denoising auxiliary task from DN-DETR is borrowed to speed up convergence. But the default noise is too large, larger than the motion between two adjacent frames, and it ends up hurting association; after the authors shrank the noise, DetA improved by 2.1 points.
Discussion
Adding things one step at a time
| Setting | Val HOTA | Test HOTA |
|---|---|---|
| MOTR | 51.7 | 54.2 |
| + Implementation tweaks | 54.8 | |
| + Passing the whole box | 57.1 | |
| + YOLOX and CrowdHuman | 63.7 | |
| + Query denoising | 64.5 | 69.9 |
| + Extra association, val set, ensemble | 73.4 |
Compiled from Table 5 of the paper, DanceTrack; each row builds on the row above.
From MOTR's 54.2 to MOTRv2's 69.9, test HOTA goes up 15.7 points. The biggest step is hooking up YOLOX and mixing in CrowdHuman for training, which adds 6.6 points on the validation set in one go.
The authors also break the YOLOX and CrowdHuman step apart:
| CrowdHuman | YOLOX | HOTA | DetA | AssA |
|---|---|---|---|---|
| 57.1 | 66.2 | 49.5 | ||
| ✓ | 60.7 | 74.8 | 49.6 | |
| ✓ | 56.7 | 73.7 | 43.9 | |
| ✓ | ✓ | 63.7 | 76.6 | 53.2 |
Compiled from Table 6 of the paper, DanceTrack validation set.
With CrowdHuman alone, detection gets better, but AssA drops 5.6 points and HOTA even dips slightly. The authors' explanation is that CrowdHuman's fake videos are made with affine transformations, so the people's "motion" is very different from dancing, and training gets pulled toward handling more difficult detections, favoring detection at the expense of association.
With YOLOX added, the situation reverses. Detection is handed to YOLOX, MOTR no longer has to struggle with detection, and the extra data from CrowdHuman can be turned into association ability; AssA rises from 49.6 to 53.2.
Taken together, these two tables illustrate the paper's core claim: detection and association conflict inside MOTR, and only by offloading detection can association be learned well.
DanceTrack

MOTRv2's HOTA is 69.9, 14.8 points above the previous best, OC-SORT. Its AssA of 59.0 is more than twenty points above OC-SORT's 38.3, and its DetA of 83.0 is also higher than OC-SORT's 80.3. It leads by a wide margin on association, and it no longer trails on detection.
The MOTRv2* in the table gets 73.4 and won the first Multiple People Tracking in Group Dance challenge. But that number also adds validation-set training, an ensemble of 4 models and a post-processing rule: if exactly one new track appears within 20 to 100 frames after a track ends, the two are treated as the same person. The rule is very practical, but it's no longer end-to-end.
BDD100K
BDD100K has 8 classes. MOTRv2 gives each class its own shared query, and on the validation set it gets mMOTA 43.6 and mIDF1 56.5, 2.4 and 2.5 points above the previous best, Unicorn.
Compared with a MOTR that is also jointly trained with BDD100K images and also passes the whole box, but doesn't use YOLOX, mMOTA is 8.1 points higher and mIDF1 8.3 points higher. YOLOX helps just as clearly in a multi-class setting.
MOT17 and MOT20
On MOT17, MOTRv2 gets HOTA 62.0, MOTA 78.6 and IDF1 75.0, much better than MOTR's 57.8, 73.4 and 68.6, but still below ByteTrack (63.1, 80.3, 77.3) and BoT-SORT (64.6, 80.6, 79.5).
MOT20 is the same. MOTRv2's HOTA is 60.3, and 61.0 after joint training with MOT17, still below the table's ByteTrack at 61.3 and OC-SORT at 62.4.
The authors attribute the gap to the amount of data. MOT17 has too few training videos to train a query-centered tracker; after adding MOT17 to the training, the gap on MOT20 narrowed somewhat.
To make good use of YOLOX's boxes in crowded scenes like MOT17 and MOT20, the authors also add "track query alignment": MOTR's predicted boxes and YOLOX's boxes are matched with the Hungarian algorithm on IoU, YOLOX's box replaces the anchor box wherever the IoU is above 0.5, and unmatched predictions are removed as false positives. Aligning the anchor boxes alone raises MOTA on the MOT17 half-validation set from 75.9 to 84.3. The propagation of the query embeddings themselves is unchanged, but this step actually brings IoU matching back in.
Limitations
The authors list three limitations:
- It needs a lot of data. It doesn't work well enough on small datasets; MOT17 is an example.
- Duplicate track queries. When two people cross paths, a track query occasionally follows the wrong person, and two track queries end up following the same person.
- Speed. On a 2080Ti, YOLOX runs at 25 frames per second and MOTR at 9.5, for only 6.9 combined.
Conclusion
MOTRv2 doesn't change MOTR's association; it hands detection off instead. An external detector's boxes become the queries' anchor boxes, and MOTR only has to adjust positions and decide who's who.
In a setting like DanceTrack, where detection is easy and association is hard, this division of labor works remarkably well, and HOTA jumps by nearly 15 points in one go.
Still, to track better on MOT17, the authors ended up using IoU matching to align the anchor boxes, and added a post-processing rule to go after the DanceTrack title. For an end-to-end tracker to get the best results, it still has to borrow a few moves from tracking-by-detection in the end.