Back to paper library
arXiv 2016Object trackingPaper notes

MOT16

Who Should Be Tracked?

Original paper

MOT16: A Benchmark for Multi-Object Tracking

Paper authors

Anton Milan · Laura Leal-Taixe · Ian Reid · Stefan Roth · Konrad Schindler

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:1603.00831
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingmot162016,
  title = {{MOT16: A Benchmark for Multi-Object Tracking}},
  author = {Anton Milan and Laura Leal-Taixe and Ian Reid and Stefan Roth and Konrad Schindler},
  year = {2016},
  eprint = {1603.00831},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1603.00831}
}
On this page

Read tracking papers for long enough, and the names MOT16 and MOT17 turn up in nearly every one. DeepSORT reports its results on MOT16, and the main tables of Tracktor, FairMOT, ByteTrack and BoT-SORT are all on MOT17.

This paper is where they come from.

In October 2014, researchers from the University of Adelaide, ETH Zurich and TU Darmstadt launched MOTChallenge. It consisted of a shared dataset, a centralized evaluation script and a leaderboard anyone could upload results to. The first version, MOT15, had 11 training videos and 11 test videos. By the time this paper was written, 47 methods had uploaded results and more than 180 users had registered. SORT's results were run on MOT15.

But after more than a year of use, MOT15's problems had surfaced as well. So Anton Milan, Laura Leal-Taixé and colleagues released a second version, MOT16.

Defining the problem

Evaluating multi-object tracking is harder than evaluating many other vision tasks. The authors give three reasons.

1. The right answer itself is hard to pin down. Is a person who is mostly occluded a target to be tracked? What about a reflection in a shop window, a person on a poster, a mannequin? Even human annotators may disagree with one another.

2. Metrics disagree. CLEAR MOT was the mainstream at the time, but the IoU threshold for deciding that "the predicted box matched the ground-truth box" ranged anywhere from 25% to 50% across papers; the evaluation code also differed in its implementation details, and results varied a lot.

3. There was no fixed training and test split. Take PETS 2009. On its easiest sequence, S2L1, tracking accuracy had long since passed 90% given good detections. But everyone used different subsets, training schemes, evaluation code and detection inputs, so the results couldn't really be compared. The authors put it bluntly. Are these sequences already too easy for current methods? Or are the methods just overfitting, or is something wrong with the evaluation?

MOTChallenge answers these questions with shared data, centralized evaluation and a public leaderboard. But the first version, MOT15, had flaws of its own:

  • Inconsistent annotation rules. The annotations for some sequences were carried over directly from various older datasets, with different standards.
  • An unbalanced distribution of crowd density between the training and test sets.
  • Some sequences were too easy and too well known (PETS09-S2L1, for example), and methods had all overfit to them.
  • The official detections were too poor, and some participants simply swapped out the detector.

The SORT paper had already demonstrated that last point. It argued that replacing MOT15's official ACF detections was exactly what lifted its tracking scores by a wide margin.

Solving the problem

MOT16 came with all-new videos, a strict set of annotation rules and a clearly specified way of evaluating.

14 new videos

Original Fig. 1. Thumbnails of MOT16's 14 videos. The top row shows the 7 training videos and the bottom row the 7 test videos, including daytime street scenes, a square at night, an indoor shopping mall, crowds viewed from above, and footage from handheld or vehicle-mounted moving cameras.
Original Fig. 1, Milan et al., from PDF page 3, arXiv license, scholarly quotation.

MOT16 has 14 videos in total, half for training and half for testing, and the test annotations aren't public. Compared with MOT15, it has nearly 3 times as many boxes, and the average crowd density is also 3 times higher.

The videos were deliberately made to differ from one another:

  • Camera: fixed in the scene, or moving: handheld, on a stroller or mounted on a car.
  • Viewpoint: looking down from high up, at pedestrian height, or from low down.
  • Conditions: sunny (with shadows and overexposure), overcast, night (lots of motion blur), indoors.
  • Frame rate: mostly 30 FPS, with some videos at 25 and 14 FPS.

According to the statistics in Table 2 of the paper, the training set has 5,316 frames, 512 tracks and 110,407 pedestrian boxes, an average of 20.8 people per frame; the test set has 5,919 frames, 830 tracks and 182,326 pedestrian boxes, an average of 30.8 people per frame.

These statistics hide a detail worth noticing. MOT16-03 in the test set is a crowd at night seen from above, with an average of 69.7 people per frame. This one sequence alone has 104,556 pedestrian boxes, 57% of the entire test set. MOTA is computed with all sequences pooled together, so performance on this one sequence has a big say in the overall score.

Who should be tracked?

Original Fig. 2. On the left is an overview of the annotation classes. The orange "Target" group is what gets evaluated, including Pedestrian (people walking, running or standing) and On vehicle (people riding bicycles or skateboards); the red "Ambiguous" group holds the ambiguous classes, including Static person (people sitting or lying down), Distractor (mannequins, posters, statues) and Reflection; the green "Other" group includes Car, Motorbike, Occluder (lamp posts, trash bins, pillars, trees, strollers) and Bicycle. On the right is an annotation example in which a person cut off by the image edge has a box that extends beyond the image.
Original Fig. 2, Milan et al., from PDF page 4, arXiv license, scholarly quotation.

Every sequence was re-annotated from scratch by researchers following the same rules, then cross-checked. At the heart of the rules is sorting "things that look like people" into three groups:

  1. Targets: every upright person, whether walking, running or standing. These are what the tracker is meant to track.
  2. Ambiguous classes: people who aren't upright (sitting, lying down), and all kinds of stand-ins for people, including reflections, posters, photos, mannequins and people behind glass.
  3. Other: moving cars, motorbikes, bicycles and strollers, plus occluders such as pillars and trees.

The second group is where this paper is really clever. A detector that sees a person on a poster will quite likely box it too; if the tracker dutifully tracks that box, is it a mistake? MOT16's answer is that such cases are neither penalized nor rewarded. During evaluation, any predicted box that overlaps a distractor, a static person, a reflection or a person on a vehicle by more than 50% is simply removed. Such a box doesn't count as a false positive, and not tracking these people doesn't count as a missed detection either.

The third group doesn't take part in evaluation, but having it annotated means it can be used for training, and also for computing how much of each pedestrian is occluded.

The boxes themselves have rules as well:

  • The box has to enclose all of the person's pixels while being as tight as possible. For a person walking sideways, the box width changes with the stride cycle; for occluded parts, the full extent has to be estimated from cues such as height, shadows and the surrounding frames; for a person cut off by the image edge, the box can extend beyond the image.
  • Annotation starts as soon as the position can be determined, roughly when a tenth of the person is visible, and ends when it can't be determined any more.
  • IDs are kept whenever possible. As long as the person is still in the frame and their path can be clearly made out, they keep the same ID. While they're occluded, annotation continues as long as simple reasoning such as constant velocity can pin down their position. If they're occluded too long for the path to be made out, they get a new ID when they reappear.
  • Every box has a visibility value. It isn't annotated by hand but computed automatically from a ground-plane assumption and the positions of all the boxes, and ranges from 0 to 1.

Finally there's a "sanity check": a pedestrian detector is run over all the videos, and any people or distractors the detector is highly confident about but the annotations missed are added.

These rules went on to have a deep influence. The visibility analysis we'll see in the Tracktor note, and the "box extends beyond the image" problem that CenterTrack and ByteTrack handle specially, both come from here.

Official detections and the data format

The authors tested several pedestrian detectors and ended up providing DPM v5 detections as the official public detections. The reason is that the off-the-shelf R-CNN of the time beat DPM on every class except "person," and pedestrian detection happened to be exactly where it lost to DPM. To keep recall up, the detection threshold was set very low.

All annotations and detections are comma-separated text files, one box per line, with nine fields: frame number, ID, the x and y of the box's top-left corner, width, height and confidence score (in annotation files, a flag for whether the box counts in evaluation); in annotation files, the last two columns are the class and the visibility. The ID in detection files is always −1, because nothing has been tracked yet.

This format is still in use today. When you see a tracker on GitHub writing its results line by line as frame,id,x,y,w,h,..., nine times out of ten it's lining up with MOTChallenge.

How evaluation is computed

Evaluation follows CLEAR MOT (MOTA, MOTP) and the track quality measures (MT, ML, FM) proposed by Wu and Nevatia. The point is to spell out every detail.

Original Fig. 4. Four cases of matching between tracker output and ground-truth trajectories. Dashed lines are ground-truth trajectories, solid lines are tracker output, colors stand for IDs, and the gray area is the matching threshold. (a) The match switches from the red track to the blue one, which counts as one ID switch. (b) The target is tracked in frames 1 and 2, interrupted and then picked up again, which counts as one fragmentation, and the new blue track also causes an ID switch. (c) The best match from frame 1 is carried all the way through, causing 5 missed detections and 4 false positives. (d) A degenerate example showing that the evaluation doesn't handle target re-identification.
Original Fig. 4, Milan et al., from PDF page 8, arXiv license, scholarly quotation.

Matching. In each frame, the Hungarian algorithm matches predicted boxes to ground-truth boxes one-to-one, with an IoU threshold of 50%. But matching isn't done independently in every frame. If a ground-truth target was matched to track jj in the previous frame and is still within the threshold of jj in this frame, that match is kept, even if there's a closer predicted box nearby.

ID switches. If the track a ground-truth target is matched to this time differs from the track it was last matched to, that counts as one ID switch (IDSW). The authors point out that a method that tracks twice as many trajectories will naturally have more ID switches too, so they also report a relative figure, ID switches divided by recall.

Fragmentation. If a ground-truth trajectory goes from "tracked" to "not tracked" and is later picked up again, that counts as one fragmentation (FM).

MOTA and MOTP. MOTA adds up missed detections, false positives and ID switches, divides the sum by the total number of ground-truth boxes and subtracts the result from 1. It ranges from negative infinity to 100%, so it can go negative when there are too many errors. MOTP is the average overlap of all matched boxes. The authors say outright that it mainly reflects the detector's localization precision and says little about the tracker's own performance.

MT and ML. A ground-truth trajectory that is tracked for more than 80% of its time counts as "mostly tracked"; under 20%, as "mostly lost." Neither metric cares whether the ID changed along the way.

All the test sequences are pooled together for computation, rather than computed separately and averaged, because the number of people differs too much from sequence to sequence.

The authors are also candid about the limits of these metrics. MOTA combines three kinds of errors into one number, which makes comparison convenient, but "it is highly debatable whether this number alone can serve as a single performance measure." The authors also state that this evaluation doesn't handle re-identification. A target that leaves the frame and comes back is treated as a new target.

Discussion

Original Table 7. Results of baseline methods on the MOT16 test set. The columns are MOTA, MOTP, false alarms per frame (FAR), MT, ML, FP, FN, ID switches, relative ID switches, fragmentations, relative fragmentations and speed (Hz). TBD has a MOTA of 33.7, ML of 54.2%, FN of 112,587, 2,418 ID switches and 1.3 Hz; CEM has 33.2, 54.4%, 114,322, 642 and 0.3 Hz; DP_NMS 32.2, 62.1%, 121,579, 972 and 212.6 Hz; SMOT 29.7, 47.7%, 107,552, 3,108 and 0.2 Hz; JPDA_M 26.2, 67.5%, 130,549, 365 and 22.2 Hz.
Original Table 7, Milan et al., from PDF page 11, arXiv license, scholarly quotation.

The authors reran five methods common at the time from their public code, covering network flow (DP_NMS), continuous energy minimization (CEM), motion similarity (SMOT), two-stage matching (TBD) and approximate joint probabilistic data association (JPDA_M).

The authors also spell out how the parameters were tuned. Starting from the defaults suggested by each method's authors, every parameter was sampled at random within [12θ,2θ][\tfrac{1}{2}\theta, 2\theta], 20 runs were done on the training set, the setting with the highest MOTA was picked, and that was run on the test set just once. The authors also state that these numbers may not be each method's best performance.

Three things in the results are worth a look:

  1. MOTA tops out in the low thirties. The best, TBD, is 33.7; the worst, JPDA_M, is 26.2.
  2. Most people get lost. Every method's ML is above 47%, and every FN is over a hundred thousand. In other words, the tracker can't rescue a person the detector didn't find, and the official DPM detections missed a lot of people.
  3. Speed differs by a factor of a thousand. DP_NMS runs at 212.6 frames per second, SMOT at only 0.2. Methods with similar accuracy can differ in speed by that much.

Hold this table up against later papers, and you can see how much the detector matters. A year later, DeepSORT switched to a set of much higher-quality private detections and reached a MOTA of 61.4 on the same MOT16 test set; a few years after that, FairMOT got 74.9. Tracking methods certainly improved, but most of the credit for the jump from the thirties all the way into the seventies belongs to the detectors.

What came after

In 2017, MOTChallenge released MOT17. The videos are exactly the same as MOT16's, but the annotations went through another round of corrections, and the official detections grew from a single DPM set to three sets, DPM, Faster R-CNN and SDP. Participants have to run once with each of the three sets and average the results. The "public detection" track we'll see in the Tracktor note is a competition under this setup.

MOT20, in 2020, pushed crowd density to the limit; according to the statistics in the ByteTrack paper, its test set averages 170 people per frame. The same year, Leal-Taixé and other MOTChallenge maintainers helped propose HOTA, a formal answer to this paper's line that it is "highly debatable" whether MOTA alone can serve as a single performance measure.

Conclusion

MOT16 gave multi-object tracking 14 new videos, a set of annotation rules rebuilt from scratch, and an evaluation that clearly defines matching, ID switches and fragmentation. Its most finely judged design choice is putting reflections, posters and seated people into ambiguous classes that are "neither penalized nor rewarded," so that trackers don't lose points for a detector's reasonable reactions.

Next time you see MOT17 numbers in a paper's table, first ask whether they used public detections or their own. Then think about how much weight that crowd at night in MOT16-03 carries in the test-set MOTA.

As for who should be tracked, ten years on, people are still using this paper's answer.