Tracking without bells and whistles
- Publication
- ICCV 2019Conference
- First submitted to arXiv
- arXiv:1903.05625
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingtracktor2019,
title = {{Tracking without bells and whistles}},
author = {Philipp Bergmann and Tim Meinhardt and Laura Leal-Taixe},
year = {2019},
eprint = {1903.05625},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1903.05625}
}
Multi-object tracking papers kept getting more complicated. Graphical models, multicuts, conditional random fields, social force models, every kind of elaborate global optimization took its turn.
Yet MOTA on MOT16 had improved by only 2.4 points in two years.
Philipp Bergmann, Tim Meinhardt and Laura Leal-Taixé at the Technical University of Munich wanted to try a different direction. Leal-Taixé is also one of the founders of MOTChallenge and knows this leaderboard inside out. They stripped every specialized part out of tracking, kept only a detector, and looked at how far it could go.
As it turned out, this tracker "without bells and whistles" beat every published method of the time on MOTA in the public detection track of all three MOTChallenge benchmarks.
That makes you wonder. If a detector alone can solve most of the tracking problem, what exactly are specialized tracking algorithms solving?
Defining the problem
SORT and DeepSORT, which we read earlier, both split tracking into two independent steps. The detector finds boxes in each frame on its own, and the tracker then works out how to link them up. Linking means predicting where an object will be in the next frame, deciding whether two boxes are the same person, and handling occlusion, missed detections and false positives.
Each of these subproblems spawned a whole family of methods. But the authors noticed that object detectors already contain a component that "predicts position."
Take Faster R-CNN. It first uses a region proposal network (RPN) to generate a large number of candidate boxes, then does two things with each one: the classification head decides whether the box contains a person, and the regression head nudges the box until it fits tightly around the object.
The regression head's job is "given a rough box, correct it to where the object really is."
What if you hand it the previous frame's box?
As long as the object doesn't move much between two frames, the previous frame's box still roughly frames the same person in this frame, and the regression head will pull it to the person's new position. The box's ID carries straight over, and the track is extended by one frame.
No Kalman filter needed, and no Hungarian algorithm either.
Solving the problem
The authors call this tracker Tracktor. It uses a Faster R-CNN trained on the MOT17Det pedestrian detection data, with a ResNet-101 plus FPN backbone.
The whole process has only two steps. Let's follow the paper's figure and see what happens at frame :

Step one: regression pushes the old box to the new position
Start with the pedestrian in dark clothes in the top left. In frame , their box is .
At frame , Tracktor puts the same coordinates directly onto the new image; that's the dashed blue box in the figure. The person has walked on a little, and the dashed box no longer quite frames them.
Next, Faster R-CNN does RoI pooling on frame 's feature map with this dashed box, exactly as it would for an ordinary candidate box, and feeds it into the regression and classification heads. The regression head outputs the corrected box , fitting tightly around the person's new position; the classification head outputs a score for whether there's still a person in the box.
The box's ID passes straight from to . That's all tracking takes here, with no matching anywhere in between.
After regression, there are two cases in which a track is ended (killed):
- The score is too low. If , there's no longer a person in the box. The object may have walked out of the frame, or been blocked by something that isn't a pedestrian, such as a pillar or a car.
- It collides with someone else. When two people cross, the boxes of both tracks may get regressed onto the same person. The authors run non-maximum suppression (NMS) over the new boxes of all tracks with an IoU threshold of , and the track that gets suppressed is ended.
Step two: detection boxes bring in new people
Tracks only continue existing objects; people who newly walk into the frame have to be found by the detector. The detector runs once over the whole of frame , giving the detections .
A detection box starts a new track only if its IoU with every existing track is below . In other words, it has to cover an object that no track explains yet. The red box in the bottom right of the figure was added this way.
The settings given in the paper's appendix are , and , with no separate tuning for any benchmark. is looser than , so existing tracks are less easily ended by NMS and new tracks are less likely to pop up, which makes tracking more stable.
Tracktor as a whole was never trained on any tracking annotations. The detector only ever learned "where the people are in each frame"; it never learned "who is who."
Two small extensions
Regression works on the premise that objects don't move much between two adjacent frames. When the camera moves a lot or the video's frame rate is low, that premise breaks down. In extreme cases, the previous frame's box may not cover the person at all in this frame.
The authors add two simple motion models:
- Camera motion compensation (CMC). The Enhanced Correlation Coefficient (ECC) is used to align two adjacent frames, so the old box is first moved along with the camera's motion before it's handed to the regression head.
- Constant velocity assumption (CVA). This is for low-frame-rate videos. Across all the experiments, only one sequence, AVG-TownCentre in MOT15, uses it.
The other extension is short-term re-identification (reID). A killed track isn't thrown away right away but kept for a while. If a new detection box shows up in the meantime, a Siamese network compares their appearance distance, and if it's below a threshold, the old ID is reattached. To avoid mistaking one person for another, the authors also require the IoU between the two boxes to be large enough, and the motion model keeps pushing these retained old boxes forward.
This reID network is TriNet with a ResNet-50 backbone, trained with a triplet loss on MOT17's tracking annotations. So strictly speaking, once reID is added, it's no longer "no tracking annotations at all," and the paper says so explicitly.
The version with CMC and reID is called Tracktor++.
Discussion
Ablation: what each extension is worth

This table holds three comparisons, and each makes one point:
- Tracking with a detector beats a detect-and-track model trained specifically for the job. D&T is trained on tracking annotations, produces short two-frame tracks and then links them offline into long ones. Its MOTA is 50.1 and its IDF1 only 24.9; Tracktor, which has never touched tracking annotations, gets 61.5 and 61.1.
- A better detector means better tracking. Tracktor-no-FPN, with FPN removed, drops from 61.5 to 57.4 MOTA. The quality of the detector directly determines the quality of the regression.
- The two extensions mainly rescue IDs. From Tracktor to Tracktor++, MOTA gains just 0.4, but ID switches fall from 1747 to 326 and IDF1 rises from 61.1 to 64.7. CMC does more than reID here: CMC alone brings ID switches down to 458. Several MOT17 sequences are shot with moving cameras, and old boxes that can't keep up with the camera's movement are one of the main reasons IDs break.
The test sets
On the test sets of the three benchmarks, Tracktor++ reaches a MOTA of 53.5 on MOT17, 54.4 on MOT16 and 44.1 on MOT15, in each case the highest among published public detection methods at the time. All three benchmarks use the same detector, trained on MOT17Det; MOT15's training data isn't used at all.
On MOT17, Tracktor++ has 12201 FP, against 33212 for the runner-up, eHAF, so it makes far fewer false positives; its IDF1 of 52.3 is slightly lower than eHAF's 54.7 and jCC's 54.5. Its MOTA lead comes mainly from fewer false positives, while on identity preservation it merely keeps up.
The really hard cases
The authors didn't stop at the leaderboard. On the MOT17 training set, they put Tracktor side by side with other top methods and compared them in groups by object visibility, object size and the length of detection gaps.

The colored bars in this figure are the fraction each method tracks at different visibilities, and the pale red bars show how many ground-truth boxes there are at each visibility.
Look at the right half first. Objects with visibility above 0.6 are tracked well by every method, and the pale red bars tell us that this region holds the most boxes. Now the left half. Of the objects with visibility below 0.3, every method tracks less than 20%, whether online or offline, with or without an appearance model, with or without interpolation.
Tracktor clearly leads between 0.3 and 0.6, because the regression head can still pull a box back around a half-hidden person; in the heavily occluded region, no method handles things well. On this basis, the authors point out that the decent MOTA scores on the test set come largely from the distribution of the data. Most of the boxes are clear to begin with.
The analysis by object size reaches the same conclusion. Among objects with visibility above 0.9, people taller than 250 pixels are tracked well by every method; the smaller the object, the worse every method does. The analysis of detection gaps shows that with the DPM and Faster R-CNN public detections, for gaps longer than 15 frames, Tracktor fills holes by regression far better than the other methods; with the better SDP detections, the methods that benefit most are actually the offline ones with interpolation. Every method's coverage drops as gaps get longer.
Where the ceiling is
Finally, the authors replace each component of Tracktor with ground truth, building a series of oracles to estimate the most each direction could gain:

A few numbers worth a look:
- Perfect reID (Oracle-reID) leaves MOTA unchanged but adds 10 points of IDF1. Re-identification only deals with recognizing people, not with whether the boxes are accurate.
- A perfect motion model plus reID (Oracle-MM-reID) adds 13.9 IDF1. The authors treat this as the ceiling for the Tracktor++ line of extensions.
- Adding linear interpolation during occlusions on top (Oracle-MM-reID-INTER) adds 15.9 IDF1 and cuts FN by 6769, but adds 3774 FP. Quite a few of the interpolated boxes land in the wrong place when the camera moves or an occlusion lasts a long time.
- Replacing everything with ground truth (Oracle-ALL) adds 10.7 MOTA, reaching 72.2. That's the absolute ceiling for this Faster R-CNN plus FPN detector.
Oracle-Kill is interesting too. Using ground truth to decide when to end tracks raises MOTA by 0.7, yet adds 129 ID switches. The longer tracks are kept alive, the more often the regression head runs into occluded, unclear boxes, and it ends up latching onto the wrong person.
The authors' proposal
The authors argue for treating Tracktor as a new starting point. Large, clearly visible objects go to frame-by-frame regression, and the remaining hard cases go to specialized methods. They suggest two directions: motion models that can "hallucinate" where objects are during long occlusions, and a shift from "tracking-by-detection" to "tracking-by-tracklet," with the detector producing short tracks directly and later methods handling the hard parts.
Looking back, both directions were taken up. CenterTrack and FairMOT, which put detection and tracking into the same network, appeared soon after. Two years later, Tim Meinhardt, one of this paper's own authors, published TrackFormer with Leal-Taixé and others, which carries the previous frame's objects into the next frame's Transformer decoder as "track queries," still following the idea of "using the previous frame's objects to find this frame's objects."
Conclusion
A trained detector, handing the previous frame's boxes to its regression head, got the highest public detection MOTA on three MOTChallenge benchmarks; add camera motion compensation and short-term reID, and ID switches drop by another 80%.
The analysis in the second half holds up to rereading better than the leaderboard does. Mostly occluded objects and very small objects weren't handled well by any method at the time, and the nice MOTA numbers on the leaderboard came mostly from boxes that were clear and easy to track to begin with.
The title promises no bells and whistles, but the authors still added two. Once they had, they also honestly admitted that the people hardest to track hadn't been tracked well by anyone yet.