TrackFormer: Multi-Object Tracking with Transformers
- Publication
- CVPR 2022Conference
- First submitted to arXiv
- arXiv:2101.02702
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingtrackformer2021,
title = {{TrackFormer: Multi-Object Tracking with Transformers}},
author = {Tim Meinhardt and Alexander Kirillov and Laura Leal-Taixe and Christoph Feichtenhofer},
year = {2021},
eprint = {2101.02702},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2101.02702}
}
Tracktor showed in 2019 that a detector's box regression head can push the previous frame's boxes straight into the current frame, so tracking needs almost no separate training. CenterTrack swapped in center points and offsets and followed the same route. The authors call this kind of method "tracking-by-regression."
But to reach top results, methods like these all end up bolting something else on. Tracktor needs re-ID and a motion model; other methods hook up graph optimization. The reason is that each box is regressed independently and locally. A box doesn't know who it is, and it doesn't know what the other tracks around it are doing.
In 2020, DETR framed object detection as set prediction. A set of learnable object queries goes through a Transformer decoder and directly outputs a set of non-duplicate boxes, with no anchor boxes and no non-maximum suppression.
The two authors of Tracktor, Tim Meinhardt and Laura Leal-Taixé of the Technical University of Munich, teamed up with Alexander Kirillov and Christoph Feichtenhofer of Facebook AI Research to carry this idea over to tracking. If a query can represent an object, then letting it follow that object into the next frame would be tracking, wouldn't it?
Defining the problem
A video contains objects. Each object has a track recording its box in every frame from when it enters the scene to when it leaves, including the frames in which it is occluded.
The authors frame multi-object tracking as a frame-by-frame set prediction problem. In every frame, the model has to output a set of boxes with identities, covering both where the objects already being tracked are now and which objects are new.
Detection, keeping identities and starting new tracks all happen in the same decoding pass. There's no separate matching step and no graph optimization, and appearance and motion don't need to be modeled explicitly. The authors call this new paradigm "tracking-by-attention."
Solving the problem

Architecture
TrackFormer's backbone is simply Deformable DETR:
- ResNet-50 extracts image features.
- A 6-layer Transformer encoder runs self-attention over the image features.
- A 6-layer Transformer decoder runs self-attention among the queries, then cross-attention to the encoded image features.
- An MLP maps each output embedding to a box and a class.
Everything that differs from DETR is in the decoder's queries.
Two kinds of queries

Object queries, as in DETR, are a fixed set of learnable embeddings responsible for discovering new objects in any frame. MOT17 has at most 52 people per frame, and the authors use 500 object queries; there need to be far more queries than objects in the scene for it to work well.
Track queries are the heart of this paper. When an object query detects an object in some frame (a classification score above 0.4), its output embedding from the decoder's last layer becomes a track query for the next frame directly, carrying an identity number with it.
In the next frame, the decoder processes queries at once. Through cross-attention, each track query looks for its object in the new image and outputs an updated box; the updated embedding is then passed on to the frame after that. Identities are passed along autoregressively like this, frame by frame, without any matching at any point.
The self-attention between the two kinds of queries matters too. Object queries can "see" the track queries, so they don't re-detect people who are already being tracked.
To make it easier for track queries to find their objects, the implementation has two more details. The reference point of a track query in deformable attention is set directly to the center of the previous frame's box, and the decoder's cross-attention looks at both the previous and current frames' feature maps, telling them apart with a temporal encoding.
Ending tracks and getting them back
A track query is removed when its classification score drops below 0.4, or when its IoU with another box exceeds 0.9. The authors found that some heavily overlapping duplicate boxes can't be handled by the decoder's self-attention, so they kept this very loose non-maximum suppression.
A removed track query isn't thrown away immediately. It keeps being decoded for another 5 frames, but produces no output during that time. If its score climbs back above 0.4, it's restored as a normal track. This is a short-term re-identification that still relies on the same attention, with no extra re-ID training.
But track queries carry position information. If an object is occluded for too long or moves too far, it can't be found again.
Training
For track queries to learn to follow objects, the model has to be trained on two adjacent frames:
- In frame , detection uses object queries only.
- Detected objects become track queries and are fed into frame along with the object queries.
Each output in frame has to be assigned to a ground-truth object or to background. Let and be the sets of ground-truth identities in the two frames. The assignment rules are:
- (present in both frames): track queries are matched directly by identity. This is a hard assignment.
- (present in the previous frame, gone in this one): the corresponding track queries have to predict background.
- (newly appeared): handed to the object queries, which use DETR's Hungarian algorithm to find the lowest-cost matching.
The matching cost combines the class probability and the difference between boxes:
The box cost combines L1 distance and generalized IoU:
Once assignment is done, the loss for each query is
Queries that aren't assigned to anything are all treated as background.
Three augmentations
Trained only on adjacent pairs of frames, the model sees tracking situations that are too simple. The authors add three augmentations:
- A wider frame gap. Frame isn't necessarily the immediately preceding frame; it's sampled from a range around , simulating camera motion and low frame rates.
- Missed detections. With probability , some track queries are dropped, so those objects have to be re-detected by object queries in frame . This is the key to training the two kinds of queries together.
- False positives. With probability , a fake track query is added for a given track query, taken from a previous-frame output embedding that was judged to be background and overlaps heavily with it. This lets the model practice removing tracks during occlusion.
Data
Transformers need lots of data, and MOT17 has only 7 training videos. Like CenterTrack, the authors use single images from CrowdHuman to simulate pairs of adjacent frames, applying random scaling and cropping of up to 20% to the same image.
The private-detection model is first trained from scratch on CrowdHuman for 85 epochs, then fine-tuned on a mix of CrowdHuman and MOT17 for 40 epochs, which takes about two days on 7 GPUs with 32GB each.
Discussion
Component ablation
| Setting | MOTA | Δ | IDF1 | Δ |
|---|---|---|---|---|
| Full TrackFormer | 71.3 | 73.4 | ||
| No CrowdHuman pretraining | 69.3 | −2.0 | 71.8 | −1.6 |
| Also no track-query re-ID | 69.2 | −0.1 | 70.4 | −1.4 |
| Also no FP augmentation | 68.4 | −0.8 | 70.0 | −0.4 |
| Also no frame-gap augmentation | 64.0 | −4.4 | 59.2 | −10.8 |
| Also no track queries | 61.0 | −3.0 | 45.1 | −14.1 |
Compiled from Table 3 of the paper, MOT17 training set split into first and second halves, private detections; each row removes one more component from the row above.
This table is cumulative: each row has one fewer component than the row above.
Of the training tricks, the frame-gap augmentation has the biggest effect. Without it, MOTA drops 4.4 points and IDF1 drops 10.8 points. Trained only on immediately adjacent frames, the model overfits quickly and never learns how to follow objects that move a lot.
The last row removes even the track queries. The model is trained only for detection and uses CenterTrack-style greedy matching on center distance instead, and IDF1 falls another 14.1 points. This row stands for the older approach of matching by hand-written rules.
MOT17 and MOT20

With private detections, TrackFormer reaches MOTA 74.1 and IDF1 68.0 on MOT17. Among methods that use only CrowdHuman as extra data, its MOTA is 5.0 points higher than the runner-up, TraDeS, and 6.3 points higher than CenterTrack. Its FN is only 108,777, more than fifty thousand fewer than CenterTrack, so its recall is much higher.
But the table also has several methods that use 6 extra tracking datasets, such as FairMOT, GRTU and TLR, and all of them have a higher IDF1. On MOT20, its MOTA ties SOTMOT, but it's nearly 9 points below ReMOT, and it trails on IDF1 too.
It runs at 7.4 frames per second, less than half the speed of CenterTrack's 17.7.
How to compare with public detections
TrackFormer isn't a method that takes detection boxes as input and associates them, so it can't use public detections as input directly. The authors use filtering instead: only object queries whose IoU with some public detection box exceeds 0.5 are allowed to start new tracks.
CenterTrack has a similar filter, but it uses center points; a public detection counts if its center falls inside the box. In the appendix, the authors compare the two filters and find that with the center-point version, three public detection sets of very different quality (DPM, FRCNN, SDP) produce almost identical results. In effect, the quality of the detections isn't reflected. The IoU version is stricter and lowers all the scores, but it's closer to the original intent of "using public detections as input."
Under IoU filtering, TrackFormer's MOTA is 62.3, on par with the best online methods; but its 4,018 ID switches are the most in the table.
Adding masks
TrackFormer can also take a segmentation head and do multi-object tracking and segmentation on MOTS20. The authors found that after training jointly with masks, IDF1 goes up by 1.2 points even when only box tracking is evaluated; the mask information helps resolve some ambiguous occlusion cases during training.
Limitations and what came after
TrackFormer is trained only on pairs of adjacent frames, so the temporal information its track queries learn is very short. The authors themselves write in the paper that methods going beyond two-frame training and inference should have a lot of potential.
MOTR, a few months later, did exactly that. It extends training to a whole short clip, passes track queries across many frames, and removes rules like track NMS and re-identification. Also, the TrackFormer numbers cited in the comparison tables of MOTR and the later MOTRv2 are MOTA 65.0 and IDF1 63.9 on MOT17. Those come from an earlier version; only after the final version was trained on adjacent frames simulated from CrowdHuman did they rise to the 74.1 and 68.0 here.
Conclusion
TrackFormer lets one query stand for one object, and once the object is detected, the query follows it from frame to frame. New objects go to object queries, existing ones to track queries, and the two can see each other inside the same decoder. That's how the matching step disappears.
Its numbers don't lead across the board, and it's slow. But it demonstrated another way of writing a tracker, and MOTR and MOTRv2 both built on it afterward.
Two years earlier, Meinhardt and Leal-Taixé used Tracktor to show that you can track very well without training for tracking at all. This time, tracking became something you train for two days on seven GPUs.