DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion
- Publication
- CVPR 2022Conference
- First submitted to arXiv
- arXiv:2111.14690
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingdancetrack2021,
title = {{DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion}},
author = {Peize Sun and Jinkun Cao and Yi Jiang and Zehuan Yuan and Song Bai and Kris Kitani and Ping Luo},
year = {2021},
eprint = {2111.14690},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2111.14690}
}
By 2021, the mainstream approach to multi-object tracking had more or less settled: detect first, then use re-ID features to link up each frame's boxes. JDE and FairMOT add an embedding branch to the detection network, QDTrack uses quasi-dense contrastive learning to train stronger appearance features, and Transformer methods like TrackFormer and MOTR also rely on the appearance information their queries carry.
Researchers from the University of Hong Kong, Carnegie Mellon University and ByteDance, including Peize Sun, Jinkun Cao, Kris Kitani and Ping Luo, questioned this trend. Could it be that appearance features work so well only because the datasets play along?
The pedestrians in MOT17 all dress differently, and they walk in regular directions at regular speeds, almost in straight lines. On data like that, recognizing people by how they look, or predicting positions with a linear motion model, both succeed easily. But switch to a different scene, say a group of dancers in identical uniforms moving quickly around a stage and crossing past one another. Can these methods still hold up?
Interestingly, Peize Sun, Yi Jiang, Zehuan Yuan and Ping Luo, who are among the authors here, had published ByteTrack with others just over a month earlier, a tracker that scores extremely high on MOT17 using only motion and IoU.
Defining the problem

The authors argue that existing datasets have two biases:
- Appearance is too easy to tell apart. Most objects look different from one another, so a re-ID model can distinguish them easily.
- Motion is too regular. Pedestrians and vehicles mostly move at close to constant velocity in straight lines, so a simple linear motion model is enough.
Trackers developed on data like this naturally come to rely more and more on appearance matching, and other cues rarely get serious consideration.
One thing to note here. Making scenes more crowded doesn't solve this problem. MOT20's crowds are extremely dense, but the pedestrians move regularly, and their relative positions and occluded regions barely change. That kind of "crowding" poses no obstacle at all to appearance matching. What the authors want is a different kind of difficulty: objects that move over a larger area, occluded regions that keep changing, and objects that cross and swap places.
Solving the problem
Collection and annotation

Group dance happens to have both kinds of difficulty at once. Dancers often wear similar or even identical outfits, and they move across large areas, strike all kinds of poses and cross paths constantly.
The authors collected publicly available videos from the internet using keywords such as "street dance," "hip-hop dance," "cheerleading" and "rhythmic gymnastics," 100 videos in all. The types include street dance, pop dance, classical dance such as ballet and tango, and large group dance, plus sports scenes such as gymnastics, Chinese kung fu and cheerleading. Most are indoors, but there are also outdoor, dimly lit and distant-camera examples, and the most crowded video has 40 people.
The annotations are boxes and identities only, with these rules:
- For partly occluded people, annotate the full-body box.
- People who are fully occluded aren't annotated; when they reappear, they keep their original identity.
The annotation tool carries the previous frame's boxes into the next frame automatically, so annotators only have to correct them. After annotation, a separate group checks the work and sends errors back to be redone.
The 100 videos are split into 40 for training, 25 for validation and 35 for testing, keeping average length, average number of boxes, and the diversity of scenes and motion roughly the same across the three. Test set annotations aren't released; they're reserved for the competition.
| MOT17 | MOT20 | DanceTrack | |
|---|---|---|---|
| Videos | 14 | 8 | 100 |
| Avg. tracks per video | 96 | 432 | 9 |
| Total tracks | 1,342 | 3,456 | 990 |
| Avg. length (s) | 35.4 | 66.8 | 52.9 |
| Total length (s) | 463 | 535 | 5,292 |
| Total images | 11,235 | 13,410 | 105,855 |
Compiled from Table 1 of the paper.
The numbers of videos and images are both about ten times those of the MOT datasets. But each video averages only 9 tracks, far fewer than MOT17. DanceTrack's difficulty isn't the number of people; it's that everyone looks alike and moves erratically.
Putting the difficulty into numbers

The authors use three statistics to quantify the two kinds of difficulty.
Appearance similarity. Use a pretrained re-ID model to extract each person's features, compute the cosine distance between every pair of people in the same frame, and average over the whole video. DanceTrack's average distance is clearly smaller than MOT17's, so different people look more alike.
IoU between adjacent frames. Take the IoU of the same person's boxes in two adjacent frames, then average over all people and all frames. A low IoU means fast motion or a low frame rate. DanceTrack is about the same as the MOT datasets, so the difficulty of this dataset comes from somewhere other than unreasonably fast motion.
Frequency of relative position switches. Whenever two people whose boxes overlap swap their left-right or up-down positions between two adjacent frames, count one switch. Objects moving at constant velocity rarely swap places; it's nonlinear motion that makes paths cross often. DanceTrack is far higher on this than MOT17 and MOT20, even higher than KITTI, a driving dataset where the camera is always moving.
Put the last two together and the people in DanceTrack don't move much from frame to frame, yet their tracks keep crossing. That's exactly where linear motion models are most likely to go wrong.
Evaluation metrics
DanceTrack uses HOTA as its main metric. Association ability is judged by AssA and IDF1, and detection quality by DetA and MOTA. The authors' reason is that MOTA leans too heavily toward detection, and HOTA is what lets you look at detection and association separately.
Discussion
Given ground-truth boxes, can trackers still track well?
| Cues | MOT17 HOTA | DT HOTA | DT AssA | DT IDF1 |
|---|---|---|---|---|
| IoU | 98.1 | 72.8 | 53.6 | 63.5 |
| IoU + motion | 96.4 | 69.4 | 54.8 | 71.3 |
| App. + IoU + motion | 95.0 | 59.7 | 43.2 | 60.5 |
| App. | 93.3 | 68.0 | 47.4 | 58.7 |
Compiled from Table 2 of the paper, validation sets of both datasets; ground-truth boxes are used directly as detection boxes. DT = DanceTrack; App. = appearance features.
The authors start with an oracle experiment: ground-truth boxes are given directly as the detections, and only the association methods are compared.
On MOT17, matching with IoU alone gives a HOTA of 98.1, nearly perfect. In other words, as long as detection is accurate enough, association on MOT17 is hardly a problem.
On DanceTrack, with the same ground-truth boxes, IoU matching gets a HOTA of only 72.8 and an AssA of 53.6. Detection is already perfect, so the bottleneck lies entirely in association.
The last two rows add appearance features, and every metric gets worse. "Appearance + IoU + motion" even has a HOTA 13 points lower than IoU alone. In a scene where everyone looks the same, appearance not only fails to help but pulls association off course.
How existing trackers perform

The authors retrained and retested all the mainstream trackers of the time on DanceTrack.
Every method's HOTA, AssA and IDF1 drop sharply on DanceTrack. ByteTrack goes from 63.1 on MOT17 to only 47.7 on DanceTrack; FairMOT falls from 59.3 to 39.7.
But the detection metrics actually improve. Every method's MOTA and DetA are higher on DanceTrack than on MOT17, with MOTA generally between 80 and 90. People on a stage are easy to detect; the trackers just can't tell who's who.
This table was updated across versions of the paper. The third version switches QDTrack's detector to YOLOX and retrains it with ByteTrack's data augmentation and thresholds, leaving the appearance-learning part completely unchanged, and the result is a HOTA of 54.2, higher than ByteTrack. MOTR's AssA of 40.2 is the highest in the table, but its DetA of 73.5 is lower than most methods. Strong association, weak detection: that combination later became the starting point for MOTRv2.
Association methods and motion models
To compare association on its own, the authors fix YOLOX as the detector and swap only the association method:
| Association | HOTA | DetA | AssA | MOTA | IDF1 |
|---|---|---|---|---|---|
| IoU | 44.7 | 79.6 | 25.3 | 87.3 | 36.8 |
| SORT | 47.8 | 74.0 | 31.0 | 88.2 | 48.3 |
| DeepSORT | 45.8 | 70.9 | 29.7 | 87.1 | 46.8 |
| MOTDT | 39.2 | 68.8 | 22.5 | 84.3 | 39.6 |
| BYTE | 47.1 | 70.5 | 31.5 | 88.2 | 51.9 |
| OC-SORT | 52.1 | 79.8 | 35.3 | 87.3 | 51.6 |
Compiled from Table 4 of the paper, DanceTrack validation set, with YOLOX as the detector.
This table shows three things.
- DeepSORT is worse than SORT. DeepSORT is just SORT plus appearance matching, and HOTA drops from 47.8 to 45.8. Appearance makes a negative contribution here.
- MOTDT is the worst. MOTDT uses tracking results to help fill in detections, but detection on DanceTrack is already very good, and this design ends up making both detection and association worse.
- OC-SORT is the best. It's a purely motion-based method that improves the Kalman filter specifically for nonlinear motion, with a HOTA of 52.1 and the highest AssA, 35.3, as well.
Another table compares motion models. With no motion at all, only IoU, HOTA is 44.7; adding a Kalman filter gives 47.8; switching to a motion model learned by an LSTM gives 51.6. When appearance is unreliable, motion models have a lot of potential.
Adding more information
Since detection isn't the problem, the authors try giving the model more information beyond boxes. DanceTrack only has box annotations, so they use CenterNet with BYTE association and train it together with other datasets:
- Segmentation masks: training together with COCO masks raises HOTA from 36.9 to 38.1; also using the masks during association brings it to 39.2.
- Human pose: training together with COCO poses raises HOTA to 40.6; also using poses during association brings it to 41.0. That's a bigger gain than masks. When most of a person is hidden, the box is unreliable, and keypoints are actually more stable.
- Depth: training together with depth from KITTI actually drops HOTA to 34.4. KITTI is mainly cars, too different from a stage. Still, taking the jointly trained model as the baseline, using depth during association adds 1.7 IDF1.
The authors conclude that representations finer than boxes (masks, poses) help, depth information has potential, and motion modeling is important.
Limitations
The authors point out two limitations themselves. This paper only proposes a dataset, not a new algorithm that pulls far ahead, and leaves the problem to future research. Also, the dataset has only box annotations, with no poses or masks, so finer-grained research has to borrow other datasets.
Afterward
DanceTrack quickly became the standard benchmark for testing association ability.
A few months later, Jinkun Cao and Kris Kitani, two of its authors, published OC-SORT, which deals specifically with nonlinear motion and occlusion. Later still, MOTRv2 and Deep OC-SORT both made DanceTrack their main stage for comparison.
Conclusion
DanceTrack doesn't propose a new tracker. It finds a scene that makes every existing method miserable: everyone wears the same clothes and jumps around.
In this scene, detection is hardly a problem; association is. Appearance features often do more harm than good, and linear motion models can't keep up with the dance steps.
When people used to talk about what makes tracking hard, the first thing that came to mind was "crowds." Each DanceTrack video averages only 9 people, and it's harder to track than MOT20's crowds of hundreds.