Back to paper library
arXiv 2021CVPR 2022Object trackingPaper notes

DanceTrack

Dressed Alike

Original paper

DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion

Paper authors

Peize Sun · Jinkun Cao · Yi Jiang …

Show all 7 authors

Peize Sun · Jinkun Cao · Yi Jiang · Zehuan Yuan · Song Bai · Kris Kitani · Ping Luo

Publication
CVPR 2022Conference
First submitted to arXiv
arXiv:2111.14690
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingdancetrack2021,
  title = {{DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion}},
  author = {Peize Sun and Jinkun Cao and Yi Jiang and Zehuan Yuan and Song Bai and Kris Kitani and Ping Luo},
  year = {2021},
  eprint = {2111.14690},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.14690}
}
On this page

By 2021, the mainstream approach to multi-object tracking had more or less settled: detect first, then use re-ID features to link up each frame's boxes. JDE and FairMOT add an embedding branch to the detection network, QDTrack uses quasi-dense contrastive learning to train stronger appearance features, and Transformer methods like TrackFormer and MOTR also rely on the appearance information their queries carry.

Researchers from the University of Hong Kong, Carnegie Mellon University and ByteDance, including Peize Sun, Jinkun Cao, Kris Kitani and Ping Luo, questioned this trend. Could it be that appearance features work so well only because the datasets play along?

The pedestrians in MOT17 all dress differently, and they walk in regular directions at regular speeds, almost in straight lines. On data like that, recognizing people by how they look, or predicting positions with a linear motion model, both succeed easily. But switch to a different scene, say a group of dancers in identical uniforms moving quickly around a stage and crossing past one another. Can these methods still hold up?

Interestingly, Peize Sun, Yi Jiang, Zehuan Yuan and Ping Luo, who are among the authors here, had published ByteTrack with others just over a month earlier, a tracker that scores extremely high on MOT17 using only motion and IoU.

Defining the problem

Original Figure 1. Frames 1, 66, 307 and 327 of the video DanceTrack0027. A group of dancers wearing identical white ballet skirts, each marked with a box of a different color and a number below it. As time goes on, dancers 1 to 9 keep swapping left-right positions and often occlude one another.
Original Figure 1, Sun et al., from PDF page 1, arXiv license, scholarly quotation.

The authors argue that existing datasets have two biases:

  • Appearance is too easy to tell apart. Most objects look different from one another, so a re-ID model can distinguish them easily.
  • Motion is too regular. Pedestrians and vehicles mostly move at close to constant velocity in straight lines, so a simple linear motion model is enough.

Trackers developed on data like this naturally come to rely more and more on appearance matching, and other cues rarely get serious consideration.

One thing to note here. Making scenes more crowded doesn't solve this problem. MOT20's crowds are extremely dense, but the pedestrians move regularly, and their relative positions and occluded regions barely change. That kind of "crowding" poses no obstacle at all to appearance matching. What the authors want is a different kind of difficulty: objects that move over a larger area, occluded regions that keep changing, and objects that cross and swap places.

Solving the problem

Collection and annotation

Original Figure 2. Thumbnails of all 100 DanceTrack videos, grouped by dance type with colored frames, including classical dance, pop dance, large group dance, street dance and sports. Four enlarged examples: (a) an outdoor scene; (b) a dimly lit scene shot from far away; (c) a large group dance; (d) gymnastics, with more extreme motion and body deformation.
Original Figure 2, Sun et al., from PDF page 4, arXiv license, scholarly quotation.

Group dance happens to have both kinds of difficulty at once. Dancers often wear similar or even identical outfits, and they move across large areas, strike all kinds of poses and cross paths constantly.

The authors collected publicly available videos from the internet using keywords such as "street dance," "hip-hop dance," "cheerleading" and "rhythmic gymnastics," 100 videos in all. The types include street dance, pop dance, classical dance such as ballet and tango, and large group dance, plus sports scenes such as gymnastics, Chinese kung fu and cheerleading. Most are indoors, but there are also outdoor, dimly lit and distant-camera examples, and the most crowded video has 40 people.

The annotations are boxes and identities only, with these rules:

  • For partly occluded people, annotate the full-body box.
  • People who are fully occluded aren't annotated; when they reappear, they keep their original identity.

The annotation tool carries the previous frame's boxes into the next frame automatically, so annotators only have to correct them. After annotation, a separate group checks the work and sends errors back to be redone.

The 100 videos are split into 40 for training, 25 for validation and 35 for testing, keeping average length, average number of boxes, and the diversity of scenes and motion roughly the same across the three. Test set annotations aren't released; they're reserved for the competition.

MOT17MOT20DanceTrack
Videos148100
Avg. tracks per video964329
Total tracks1,3423,456990
Avg. length (s)35.466.852.9
Total length (s)4635355,292
Total images11,23513,410105,855

Compiled from Table 1 of the paper.

The numbers of videos and images are both about ten times those of the MOT datasets. But each video averages only 9 tracks, far fewer than MOT17. DanceTrack's difficulty isn't the number of people; it's that everyone looks alike and moves erratically.

Putting the difficulty into numbers

Original Figure 3. A comparison of three statistics. (a) The cosine distance between re-ID features, with each bar one video; MOT17 averages about 0.28 and DanceTrack about 0.20, and a smaller distance means different people look more alike. (b) The average IoU between adjacent frames; MOT17, MOT20, KITTI and DanceTrack all fall between 0.8 and 0.96, with DanceTrack at about 0.9. (c) The frequency of relative position switches; MOT17 is almost 0, MOT20 about 0.007, KITTI about 0.014, and DanceTrack the highest, at about 0.022.
Original Figure 3, Sun et al., from PDF page 5, arXiv license, scholarly quotation.

The authors use three statistics to quantify the two kinds of difficulty.

Appearance similarity. Use a pretrained re-ID model to extract each person's features, compute the cosine distance between every pair of people in the same frame, and average over the whole video. DanceTrack's average distance is clearly smaller than MOT17's, so different people look more alike.

IoU between adjacent frames. Take the IoU of the same person's boxes in two adjacent frames, then average over all people and all frames. A low IoU means fast motion or a low frame rate. DanceTrack is about the same as the MOT datasets, so the difficulty of this dataset comes from somewhere other than unreasonably fast motion.

Frequency of relative position switches. Whenever two people whose boxes overlap swap their left-right or up-down positions between two adjacent frames, count one switch. Objects moving at constant velocity rarely swap places; it's nonlinear motion that makes paths cross often. DanceTrack is far higher on this than MOT17 and MOT20, even higher than KITTI, a driving dataset where the camera is always moving.

Put the last two together and the people in DanceTrack don't move much from frame to frame, yet their tracks keep crossing. That's exactly where linear motion models are most likely to go wrong.

Evaluation metrics

DanceTrack uses HOTA as its main metric. Association ability is judged by AssA and IDF1, and detection quality by DetA and MOTA. The authors' reason is that MOTA leans too heavily toward detection, and HOTA is what lets you look at detection and association separately.

Discussion

Given ground-truth boxes, can trackers still track well?

CuesMOT17 HOTADT HOTADT AssADT IDF1
IoU98.172.853.663.5
IoU + motion96.469.454.871.3
App. + IoU + motion95.059.743.260.5
App.93.368.047.458.7

Compiled from Table 2 of the paper, validation sets of both datasets; ground-truth boxes are used directly as detection boxes. DT = DanceTrack; App. = appearance features.

The authors start with an oracle experiment: ground-truth boxes are given directly as the detections, and only the association methods are compared.

On MOT17, matching with IoU alone gives a HOTA of 98.1, nearly perfect. In other words, as long as detection is accurate enough, association on MOT17 is hardly a problem.

On DanceTrack, with the same ground-truth boxes, IoU matching gets a HOTA of only 72.8 and an AssA of 53.6. Detection is already perfect, so the bottleneck lies entirely in association.

The last two rows add appearance features, and every metric gets worse. "Appearance + IoU + motion" even has a HOTA 13 points lower than IoU alone. In a scene where everyone looks the same, appearance not only fails to help but pulls association off course.

How existing trackers perform

Original Table 3. A comparison of nine trackers on the MOT17 and DanceTrack test sets, with columns HOTA, DetA, AssA, MOTA and IDF1. On MOT17, HOTA ranges from 52.2 to 63.2; on DanceTrack, HOTA ranges from 39.7 to 55.1 and AssA from 22.6 to 40.2, yet MOTA is between 79.7 and 92.0 throughout, higher than on MOT17. ByteTrack has a HOTA of 63.1 on MOT17 and 47.7 on DanceTrack; QDTrack has 54.2 on DanceTrack; MOTR has 54.2, with the highest AssA at 40.2; OC-SORT has the highest, 55.1.
Original Table 3, Sun et al., from PDF page 7, arXiv license, scholarly quotation.

The authors retrained and retested all the mainstream trackers of the time on DanceTrack.

Every method's HOTA, AssA and IDF1 drop sharply on DanceTrack. ByteTrack goes from 63.1 on MOT17 to only 47.7 on DanceTrack; FairMOT falls from 59.3 to 39.7.

But the detection metrics actually improve. Every method's MOTA and DetA are higher on DanceTrack than on MOT17, with MOTA generally between 80 and 90. People on a stage are easy to detect; the trackers just can't tell who's who.

This table was updated across versions of the paper. The third version switches QDTrack's detector to YOLOX and retrains it with ByteTrack's data augmentation and thresholds, leaving the appearance-learning part completely unchanged, and the result is a HOTA of 54.2, higher than ByteTrack. MOTR's AssA of 40.2 is the highest in the table, but its DetA of 73.5 is lower than most methods. Strong association, weak detection: that combination later became the starting point for MOTRv2.

Association methods and motion models

To compare association on its own, the authors fix YOLOX as the detector and swap only the association method:

AssociationHOTADetAAssAMOTAIDF1
IoU44.779.625.387.336.8
SORT47.874.031.088.248.3
DeepSORT45.870.929.787.146.8
MOTDT39.268.822.584.339.6
BYTE47.170.531.588.251.9
OC-SORT52.179.835.387.351.6

Compiled from Table 4 of the paper, DanceTrack validation set, with YOLOX as the detector.

This table shows three things.

  • DeepSORT is worse than SORT. DeepSORT is just SORT plus appearance matching, and HOTA drops from 47.8 to 45.8. Appearance makes a negative contribution here.
  • MOTDT is the worst. MOTDT uses tracking results to help fill in detections, but detection on DanceTrack is already very good, and this design ends up making both detection and association worse.
  • OC-SORT is the best. It's a purely motion-based method that improves the Kalman filter specifically for nonlinear motion, with a HOTA of 52.1 and the highest AssA, 35.3, as well.

Another table compares motion models. With no motion at all, only IoU, HOTA is 44.7; adding a Kalman filter gives 47.8; switching to a motion model learned by an LSTM gives 51.6. When appearance is unreliable, motion models have a lot of potential.

Adding more information

Since detection isn't the problem, the authors try giving the model more information beyond boxes. DanceTrack only has box annotations, so they use CenterNet with BYTE association and train it together with other datasets:

  • Segmentation masks: training together with COCO masks raises HOTA from 36.9 to 38.1; also using the masks during association brings it to 39.2.
  • Human pose: training together with COCO poses raises HOTA to 40.6; also using poses during association brings it to 41.0. That's a bigger gain than masks. When most of a person is hidden, the box is unreliable, and keypoints are actually more stable.
  • Depth: training together with depth from KITTI actually drops HOTA to 34.4. KITTI is mainly cars, too different from a stage. Still, taking the jointly trained model as the baseline, using depth during association adds 1.7 IDF1.

The authors conclude that representations finer than boxes (masks, poses) help, depth information has potential, and motion modeling is important.

Limitations

The authors point out two limitations themselves. This paper only proposes a dataset, not a new algorithm that pulls far ahead, and leaves the problem to future research. Also, the dataset has only box annotations, with no poses or masks, so finer-grained research has to borrow other datasets.

Afterward

DanceTrack quickly became the standard benchmark for testing association ability.

A few months later, Jinkun Cao and Kris Kitani, two of its authors, published OC-SORT, which deals specifically with nonlinear motion and occlusion. Later still, MOTRv2 and Deep OC-SORT both made DanceTrack their main stage for comparison.

Conclusion

DanceTrack doesn't propose a new tracker. It finds a scene that makes every existing method miserable: everyone wears the same clothes and jumps around.

In this scene, detection is hardly a problem; association is. Appearance features often do more harm than good, and linear motion models can't keep up with the dance steps.

When people used to talk about what makes tracking hard, the first thing that came to mind was "crowds." Each DanceTrack video averages only 9 people, and it's harder to track than MOT20's crowds of hundreds.