Back to paper library
arXiv 2022CVPR 2023Object trackingPaper notes

MOTRv2

Leave Detection to the Experts

Original paper

MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors

Paper authors

Yuang Zhang · Tiancai Wang · Xiangyu Zhang

Publication
CVPR 2023Conference
First submitted to arXiv
arXiv:2211.09791
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingmotrv22022,
  title = {{MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors}},
  author = {Yuang Zhang and Tiancai Wang and Xiangyu Zhang},
  year = {2022},
  eprint = {2211.09791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.09791}
}
On this page

MOTR made multi-object tracking fully end-to-end: track queries are passed down frame by frame, new objects are handed to detect queries, and no matching or post-processing is needed.

On DanceTrack, MOTR's association ability was the best of its time, with an AssA of 40.2, higher than methods that match by motion, like ByteTrack and OC-SORT. But its detection wasn't good. Its DetA on DanceTrack was only 73.5, well below OC-SORT's 80.3; on MOT17, its MOTA of 73.4 was nearly 7 points behind ByteTrack's 80.3.

Yuang Zhang of Shanghai Jiao Tong University and Tiancai Wang and Xiangyu Zhang of MEGVII, members of the original MOTR team, think the problem is that the same decoder has to learn two things at once. Detection and association pull against each other inside MOTR, and detection ends up poorly learned.

So why not bring in an off-the-shelf, very strong detector to help?

Defining the problem

The most direct way to hook an external detector up to MOTR is to copy TransTrack: have the track queries predict boxes, then match them to the detector's boxes using IoU.

The authors tried it. Detection improved only slightly, and it went back down the old road of "predict first, then match," losing MOTR's end-to-end nature.

What the authors want is something like tracking-by-detection, taking detection results as input so that MOTR only has to concentrate on learning association. How to "feed" detection results into a query-centered Transformer is the problem this paper sets out to solve.

Solving the problem

Original Figure 2. The MOTRv2 architecture. In each frame, YOLOX first produces candidate boxes, which are turned into proposal queries (ProposalQ), replacing MOTR's original detect queries; track queries (TrackQ) passed on from the previous frame are responsible for objects already being tracked. The two kinds of queries are concatenated and fed into MOTR together with the image features, which outputs predictions frame by frame, and the queries corresponding to the predictions are passed on to the next frame.
Original Figure 2, Zhang et al., from PDF page 2, arXiv license, scholarly quotation.

MOTRv2 consists of an off-the-shelf YOLOX detector and a version of MOTR rewritten with anchor-based queries.

Anchor-based queries

The key step comes from the idea behind detectors like DAB-DETR. A query isn't just an abstract embedding; it also carries an explicit anchor box (center, width, height). The decoder only has to predict an offset relative to the anchor box, which makes detection much easier.

The authors rewrite both MOTR's detect queries and its track queries in this anchor-based form. The original learnable positional embeddings are replaced by sine-cosine encodings of the anchor box coordinates.

That way, an external detector's boxes can be used directly as anchor boxes, and the detection prior gets plugged into MOTR.

Proposal queries

Original Figure 3. How proposal queries are generated and passed across frames. Orange is the generation path: the confidence scores of YOLOX candidate boxes are sine-cosine encoded and added to the shared query q_s to form the proposal queries, while the candidate boxes themselves serve as anchor boxes P. Blue is the propagation path: the decoder (N layers of self-attention plus deformable attention) outputs offsets Δx, Δy, Δw, Δh relative to the anchor boxes, giving this frame's predicted boxes; the predicted boxes are passed on to the next frame together with the updated track queries q_tr and concatenated with the new proposal queries.
Original Figure 3, Zhang et al., from PDF page 4, arXiv license, scholarly quotation.

In each frame, YOLOX produces NtN_t candidate boxes, each with a center (xt,yt)(x_t, y_t), width and height (wt,ht)(w_t, h_t) and a confidence score sts_t. To miss as few objects as possible, any box with a confidence score above 0.05 is kept.

Proposal queries are produced in three steps:

  1. A learnable shared query qsq_s of size 1×D1 \times D is copied into Nt×DN_t \times D.
  2. Each candidate box's confidence score sts_t is encoded with sine-cosine encoding into a DD-dimensional vector.
  3. The two are added together to give this frame's proposal queries; the candidate boxes themselves serve as their anchor boxes.

All proposal queries share the same embedding, and they differ from one another only through their anchor box positions and confidence scores. In addition, 10 learnable anchor boxes are kept to pick up objects YOLOX misses.

The authors also compared whether to encode the confidence score at all. Without scores, validation HOTA is 63.0; with a linear projection, 63.4; with sine-cosine encoding, 63.7. The confidence score is useful information for MOTR.

Passing boxes along

The first frame has only new objects. The proposal queries go through the decoder, which outputs offsets relative to the YOLOX boxes; added back, these give the predicted boxes Y^0\hat{Y}_0, and the updated queries become track queries qtr,0q_{tr,0}.

In every frame after that, the previous frame's track queries qtr,t−1q_{tr,t-1} are concatenated with this frame's proposal queries qp,tq_{p,t}; the previous frame's predicted boxes Y^t−1\hat{Y}_{t-1} are likewise concatenated with this frame's YOLOX boxes PtP_t, serving as the anchor boxes for all the queries.

That settles the division of labor. Proposal queries are responsible only for objects that are new or were previously missed, and track queries are responsible for relocating objects already being tracked. Coordination between the two is left to the decoder's self-attention. Proposal queries need to look at the track queries to avoid re-detecting someone who is already being tracked; track queries can also consult the YOLOX boxes to get their positions more accurate.

Original Figure 4. (a) Frame 100 of DanceTrack video 5: the boxes predicted by MOTR's track queries (brown) and the YOLOX candidate boxes (thick white) almost completely overlap, with four dancers numbered 1 to 4. (b) The query self-attention map, with 4 proposal queries and 4 track queries on both axes; there is clearly high attention between the proposal query and the track query of the same person.
Original Figure 4, Zhang et al., from PDF page 4, arXiv license, scholarly quotation.

The authors plot the self-attention map. Attention between the proposal query and the track query of the same person is clearly higher, so the two really are exchanging information.

The original MOTR passes along only the center point as a reference point, while MOTRv2 passes along the whole box. The authors compared the two: measured by AssA on the DanceTrack validation set, passing only the center point gives 49.2, and passing the whole box raises it to 53.1. The width and height of the previous frame's box help association a lot.

Training

During training, each short clip has 5 frames, with the interval between frames chosen at random from 1 to 10.

The original MOTR artificially inserts fake track queries or randomly drops real ones to simulate false positives and missed detections. MOTRv2 no longer does this; it simply passes on track queries whose confidence score is above 0.5. The false positives and missed detections that arise naturally during training are the most realistic augmentation there is.

On DanceTrack and MOT17, YOLOX itself isn't trained further; the weights officially released by DanceTrack and ByteTrack are used directly, and the candidate boxes for all images are computed before training.

Two more training tricks:

  • Mixing in CrowdHuman. As in MOTR, single CrowdHuman images are turned into fake 5-frame videos and trained together with DanceTrack.
  • Query denoising. The denoising auxiliary task from DN-DETR is borrowed to speed up convergence. But the default noise is too large, larger than the motion between two adjacent frames, and it ends up hurting association; after the authors shrank the noise, DetA improved by 2.1 points.

Discussion

Adding things one step at a time

SettingVal HOTATest HOTA
MOTR51.754.2
+ Implementation tweaks54.8
+ Passing the whole box57.1
+ YOLOX and CrowdHuman63.7
+ Query denoising64.569.9
+ Extra association, val set, ensemble73.4

Compiled from Table 5 of the paper, DanceTrack; each row builds on the row above.

From MOTR's 54.2 to MOTRv2's 69.9, test HOTA goes up 15.7 points. The biggest step is hooking up YOLOX and mixing in CrowdHuman for training, which adds 6.6 points on the validation set in one go.

The authors also break the YOLOX and CrowdHuman step apart:

CrowdHumanYOLOXHOTADetAAssA
57.166.249.5
✓60.774.849.6
✓56.773.743.9
✓✓63.776.653.2

Compiled from Table 6 of the paper, DanceTrack validation set.

With CrowdHuman alone, detection gets better, but AssA drops 5.6 points and HOTA even dips slightly. The authors' explanation is that CrowdHuman's fake videos are made with affine transformations, so the people's "motion" is very different from dancing, and training gets pulled toward handling more difficult detections, favoring detection at the expense of association.

With YOLOX added, the situation reverses. Detection is handed to YOLOX, MOTR no longer has to struggle with detection, and the extra data from CrowdHuman can be turned into association ability; AssA rises from 49.6 to 53.2.

Taken together, these two tables illustrate the paper's core claim: detection and association conflict inside MOTR, and only by offloading detection can association be learned well.

DanceTrack

Original Table 1. Comparison on the DanceTrack test set, with columns HOTA, DetA, AssA, MOTA and IDF1. Existing methods range in HOTA from 39.7 (FairMOT) to 55.1 (OC-SORT), and the highest AssA is MOTR's 40.2. MOTRv2 has HOTA 69.9, DetA 83.0, AssA 59.0, MOTA 91.9 and IDF1 71.7; MOTRv2*, with extra association, validation-set training and a model ensemble, has 73.4, 83.7, 64.4, 92.1 and 76.0.
Original Table 1, Zhang et al., from PDF page 5, arXiv license, scholarly quotation.

MOTRv2's HOTA is 69.9, 14.8 points above the previous best, OC-SORT. Its AssA of 59.0 is more than twenty points above OC-SORT's 38.3, and its DetA of 83.0 is also higher than OC-SORT's 80.3. It leads by a wide margin on association, and it no longer trails on detection.

The MOTRv2* in the table gets 73.4 and won the first Multiple People Tracking in Group Dance challenge. But that number also adds validation-set training, an ensemble of 4 models and a post-processing rule: if exactly one new track appears within 20 to 100 frames after a track ends, the two are treated as the same person. The rule is very practical, but it's no longer end-to-end.

BDD100K

BDD100K has 8 classes. MOTRv2 gives each class its own shared query, and on the validation set it gets mMOTA 43.6 and mIDF1 56.5, 2.4 and 2.5 points above the previous best, Unicorn.

Compared with a MOTR that is also jointly trained with BDD100K images and also passes the whole box, but doesn't use YOLOX, mMOTA is 8.1 points higher and mIDF1 8.3 points higher. YOLOX helps just as clearly in a multi-class setting.

MOT17 and MOT20

On MOT17, MOTRv2 gets HOTA 62.0, MOTA 78.6 and IDF1 75.0, much better than MOTR's 57.8, 73.4 and 68.6, but still below ByteTrack (63.1, 80.3, 77.3) and BoT-SORT (64.6, 80.6, 79.5).

MOT20 is the same. MOTRv2's HOTA is 60.3, and 61.0 after joint training with MOT17, still below the table's ByteTrack at 61.3 and OC-SORT at 62.4.

The authors attribute the gap to the amount of data. MOT17 has too few training videos to train a query-centered tracker; after adding MOT17 to the training, the gap on MOT20 narrowed somewhat.

To make good use of YOLOX's boxes in crowded scenes like MOT17 and MOT20, the authors also add "track query alignment": MOTR's predicted boxes and YOLOX's boxes are matched with the Hungarian algorithm on IoU, YOLOX's box replaces the anchor box wherever the IoU is above 0.5, and unmatched predictions are removed as false positives. Aligning the anchor boxes alone raises MOTA on the MOT17 half-validation set from 75.9 to 84.3. The propagation of the query embeddings themselves is unchanged, but this step actually brings IoU matching back in.

Limitations

The authors list three limitations:

  • It needs a lot of data. It doesn't work well enough on small datasets; MOT17 is an example.
  • Duplicate track queries. When two people cross paths, a track query occasionally follows the wrong person, and two track queries end up following the same person.
  • Speed. On a 2080Ti, YOLOX runs at 25 frames per second and MOTR at 9.5, for only 6.9 combined.

Conclusion

MOTRv2 doesn't change MOTR's association; it hands detection off instead. An external detector's boxes become the queries' anchor boxes, and MOTR only has to adjust positions and decide who's who.

In a setting like DanceTrack, where detection is easy and association is hard, this division of labor works remarkably well, and HOTA jumps by nearly 15 points in one go.

Still, to track better on MOT17, the authors ended up using IoU matching to align the anchor boxes, and added a post-processing rule to go after the DanceTrack title. For an end-to-end tracker to get the best results, it still has to borrow a few moves from tracking-by-detection in the end.