Back to paper library
arXiv 2020Object trackingPaper notes

MOT20

Packed Like Sardines

Original paper

MOT20: A benchmark for multi object tracking in crowded scenes

Paper authors

Patrick Dendorfer · Hamid Rezatofighi · Anton Milan …

Show all 9 authors

Patrick Dendorfer · Hamid Rezatofighi · Anton Milan · Javen Shi · Daniel Cremers · Ian Reid · Stefan Roth · Konrad Schindler · Laura Leal-Taixé

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2003.09003
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingmot202020,
  title = {{MOT20: A benchmark for multi object tracking in crowded scenes}},
  author = {Patrick Dendorfer and Hamid Rezatofighi and Anton Milan and Javen Shi and Daniel Cremers and Ian Reid and Stefan Roth and Konrad Schindler and Laura Leal-Taixé},
  year = {2020},
  eprint = {2003.09003},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.09003}
}
On this page

If you've read the main tables of ByteTrack, OC-SORT or StrongSORT, you've definitely seen the MOT20 column. It usually comes right after MOT17, its numbers are generally lower, and the speeds are much slower.

The MOT20 paper itself is only 7 pages long, and most of it explains the annotation rules and evaluation, which are nearly identical to MOT16 four years earlier. What's really new is 8 extremely crowded videos.

The authors are the team that maintains MOTChallenge, from the Technical University of Munich, the University of Adelaide, ETH Zurich and TU Darmstadt. According to the paper's figures, since MOTChallenge launched in 2014 it has had more than 1,000 users, and its 5 challenges have used 44 videos and 2.7 million boxes in total. The MOT20 videos were first used for a workshop challenge in 2019, and this paper is the full description that came afterward.

Defining the problem

MOT15 through MOT17 gave multi-object tracking clean data and consistent evaluation. But in the authors' experience, most methods can end up overly tuned to one particular dataset, and when moved to a completely different setting they can't cope without heavy changes to their parameters, or even a different model.

Crowding is one setting that hadn't been tested properly yet. With more people, boxes get smaller and occlude one another, detectors start missing people, and the tracker has to tell who is who in a big crowd of similar-looking boxes. The authors wanted a set of videos that would push methods "to their limits."

Solving the problem

8 videos, 3 scenes

Original Fig. 1. The 8 videos of MOT20 come from 3 scenes. The top row is the training videos and the bottom row the test videos. Scene 1 is a brightly lit indoor hall, with two training videos and one test video; scene 2 is an outdoor square at night with an extremely dense crowd, with two training videos and one test video; scene 3 is an outdoor street in daytime that appears only in the test set, with two videos.
Original Fig. 1, Dendorfer et al., from PDF page 2, arXiv license, scholarly quotation.

The 8 videos are shot in 3 scenes, 4 for training and 4 for testing, all filmed from above. Scenes 1 and 2 appear in both training and testing, and scene 3 appears only in testing. The authors deliberately hold out an unseen scene to test whether methods generalize to new environments. The test annotations aren't public, so methods can't be tuned to fit particular videos too closely.

The statistics for each video:

#SceneResolutionLen.IDsBoxesPer frame
01Indoor1920×10804297419,87046.3
02Indoor1920×10802,782270154,74255.6
03Outdoor, night1173×8802,405702313,658130.4
05Outdoor, night1654×10803,3151,169646,344195.0
04Outdoor, night1545×10802,080669274,084131.8
06Outdoor, day1920×7341,008271132,757131.7
07Indoor1920×108058511133,10156.6
08Outdoor, day1920×73480619177,48496.1

Compiled from Table 1 of the paper; test videos are in bold. Len. is the length in frames; all videos run at 25 frames per second.

A few things in this table are worth a second look.

First, the videos are short. The training videos add up to under 6 minutes and the test videos to under 3, but the training set has 1.13 million pedestrian boxes, and the test set another 520,000.

Second, the dataset is dominated by a handful of videos. MOT20-05 alone has 646,000 boxes, 57% of the whole training set; in the test set, MOT20-04 makes up 53%. MOTChallenge strings all the test videos together and computes a single MOTA, so performance on MOT20-04, this one night-time square, more or less decides the test score. The MOT16 note mentioned that MOT16-03 is in the same situation.

Third, the unseen scene 3 (MOT20-06 and 08) has a resolution of 1920×734, an aspect ratio different from all the other videos.

How many people per frame?

The paper and later citations have given several numbers for "how crowded." The paper's main text says density can reach 246 people per frame; the ByteTrack paper writes that the test set averages 170 people per image; and the test-set average in the table above is 115.5 people per frame.

The difference between these numbers lies in what gets counted. MOT20 annotates not only pedestrians but also the following classes:

ClassCount
Pedestrian1,652,040
Static person335,891
Occluder on the ground87,786
Non-motorized vehicle22,365
Crowd4,303
Total2,102,385

Compiled from Table 2 of the paper.

Adding up annotations of every class in the test set gives 765,465, which divided by 4,479 frames is about 170.9 per frame. That's the often-cited 170. Only pedestrians are actually used for evaluation, at 115.5 per frame.

As for 246, the main text calls it the average density in one place, but doesn't explain how it was computed; the highest per-video average in the table is MOT20-05's 195 people, so 246 is more likely the maximum for a single frame. Whichever number you use, MOT20 is far more crowded than MOT17.

Annotation rules

Annotation follows the MOT16 rules and sorts what's in the frame into three groups:

  1. Moving pedestrians. Everyone recognizable as a pedestrian who is moving. Someone briefly bending down to pick something up, or crouching to talk to a child, still counts in this group. This is the group trackers are supposed to track.
  2. Human figures with an ambiguous class. People sitting, lying down or standing in place for the entire video, as well as artificial representations of people (like people on a poster). In evaluation, tracking them earns nothing, and missing them costs nothing.
  3. Vehicles and occluders. These aren't part of the evaluation; they're provided for users to train on, or to compute how occluded pedestrians are. Parked cars and bicycles are annotated only when they block a pedestrian.
Original Fig. 4. Examples of MOT20 annotation classes. From left to right: two pedestrians (ped 6, ped 7), a static person (static_person), two people riding bicycles (non_mot_vhcl, non-motorized vehicle), and an information kiosk that completely blocks what's behind it (occluder_full). Each box is labeled with its class, ID and box height.
Original Fig. 4, Dendorfer et al., from PDF page 7, arXiv license, scholarly quotation.

The "neither penalized nor rewarded" group matters especially in MOT20. There are 336,000 static person annotations in total, one fifth as many as the pedestrian boxes. In the test video MOT20-08, there are 52,998 static people and 77,484 pedestrians; in other words, roughly 4 out of every 10 people in the frame are outside the scope of evaluation.

A detector, of course, doesn't care whether a person is walking. A tracker that relies on detections may well track these motionless people perfectly; if they counted as false positives, the tracker would be wrongly penalized. Conversely, a tracker that relies on motion cues and doesn't track seated people shouldn't be penalized for misses either.

The tracking target is also defined concretely, as upright pedestrians you could walk straight to along the line of sight, with no physical barrier in between. Reflections, people behind glass, and people riding bicycles or sitting in vehicles don't count.

Public detections

As with MOT17, MOT20 provides a set of public detections so everyone can compare tracking algorithms on the same input. This time it's a Faster R-CNN with a ResNet-101 backbone, trained on the MOT20 training videos. The MOT20 challenge initially accepted only results using the public detections; only when it opened for regular submissions did it also accept people's own detectors.

How this detector does on the test videos shows exactly why the authors held out an unseen scene:

VideoSceneAPRecallPrecisionFP per frame
04Night square (seen)0.6369.798.01.55
07Indoor (seen)0.7883.692.51.89
06Day street (unseen)0.4357.974.412.64
08Day street (unseen)0.3855.261.613.93

Compiled from Table 6 of the paper, test videos.

In the seen scenes, precision is above 90% everywhere, with fewer than 2 false positives per frame. On the unseen daytime street, precision drops to 60–70%, false positives per frame jump to 12 to 14, and AP is down to around 0.4.

These public detections effectively hand the generalization failure straight to the tracker. A tracker using the public detections has to deal with a large number of fake boxes on MOT20-06 and 08 from the very start.

The detection statistics also show how small the boxes are. In the four test videos, the shortest detection boxes are only 21 to 37 pixels tall; the training video MOT20-03 even has boxes as short as 16 pixels.

File format and evaluation

MOT20's files are simple CSV, one object per line, with 9 fields in order: frame number, ID, the x and y of the box's top-left corner, width, height, confidence score, class and visibility. In detection files the ID is set to −1; in ground-truth files the confidence column is used as a flag instead, with 0 meaning the entry isn't evaluated. Coordinates start from 1.

Evaluation also follows MOT16. A match needs an IoU of at least 0.5, and the metrics are CLEAR MOT plus Wu and Nevatia's track quality metrics (MT, ML, FM). The authors specifically note that the ID switch definition here is stricter than the original CLEAR MOT, counting a switch whenever the track a real target is matched to this time differs from its previous match. All test videos are strung together for the computation, rather than averaged per video.

For crowded scenes, MOT20 also adds a rule that MOT17 doesn't have. Result boxes that overlap more than 75% with classes such as static people, reflections, people on vehicles and distractors are removed from the results; the paper puts this rule in the evaluation steps of the detection challenge.

MOTA is still the main metric, and as usual the authors add that whether this number can serve on its own as the sole performance measure is "highly debatable." To reveal whether a method has been tuned to suit only a few videos, the leaderboard also lists the standard deviation of MOTA across videos.

The paper itself has no tracker results at all, only the detector performance above. For how trackers do on MOT20, you have to look at later papers and the leaderboard.

Discussion

Pulling together the places MOT20 shows up in the notes on this site, what this set of videos tests comes down to roughly four things.

Crowding slows down methods that need appearance features. FairMOT drops to 13.2 FPS on MOT20, and StrongSORT manages only 1.5 FPS. With a hundred-plus people per frame, every one of them needs an appearance feature extracted and compared once against every track.

Crowding makes the error buildup of motion models more visible. The gap between OC-SORT and ByteTrack is wider on MOT20 than on MOT17. Occlusions are frequent, and once someone is blocked it lasts several frames; with no observation to correct it, the Kalman filter's error just keeps accumulating.

Fixed cameras. All MOT20 videos come from stationary cameras, so camera motion compensation is of no use here. The Deep OC-SORT ablation shows that adding camera motion compensation gives a clear improvement on MOT17, but none on MOT20.

Crowded and hard to tell apart are two different things. The DanceTrack authors specifically point out that although MOT20 has a lot of people, pedestrians move very regularly, and their relative positions and occluded regions barely change, so this kind of crowding isn't an obstacle to appearance matching. DanceTrack therefore sets its questions from a different direction: few people, but they look the same and move erratically.

Conclusion

MOT20 keeps MOTChallenge's existing rules and swaps in 8 extremely crowded videos, with over a hundred pedestrians per frame on average, the smallest boxes only a dozen or so pixels tall, and a daytime street that appears only in testing. The paper itself is short, with no method and no tracking results, but it turned "crowding" into a column every later tracking paper has to report.

When reading its numbers, keep in mind that the test score is mostly decided by MOT20-04, a single video of a square at night, and that the oft-quoted "170 people per frame" includes classes that aren't evaluated, while the pedestrians actually evaluated number 115.5 per frame.