MOTR: End-to-End Multiple-Object Tracking with Transformer
- Publication
- ECCV 2022Conference
- First submitted to arXiv
- arXiv:2105.03247
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingmotr2021,
title = {{MOTR: End-to-End Multiple-Object Tracking with Transformer}},
author = {Fangao Zeng and Bin Dong and Yuang Zhang and Tiancai Wang and Xiangyu Zhang and Yichen Wei},
year = {2021},
eprint = {2105.03247},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2105.03247}
}
Every tracker we've read so far, no matter how much deep learning it uses, ends with a hand-written association step.
SORT uses IoU plus the Hungarian algorithm. DeepSORT adds appearance distance and a matching cascade. FairMOT keeps two-stage matching, and CenterTrack works backward from offsets and then matches greedily. Motion comes from a Kalman filter or learned offsets, appearance from re-ID features, and how the two are combined and where the thresholds sit are all decided by people. This post-processing sits in the middle, and temporal information can't flow through it end to end.
In 2020, DETR recast object detection as set prediction. A set of learnable object queries interacts with image features, each query outputs one box, and bipartite matching pairs them one-to-one with ground-truth boxes. It doesn't even need non-maximum suppression.
If detection can do without post-processing, what about tracking?
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang and colleagues at MEGVII and Shanghai Jiao Tong University propose MOTR, an attempt to build a tracker that needs no hand-written matching from start to finish.
Defining the problem
The authors take their inspiration from machine translation. A translation model uses a hidden state to represent the current context, and the decoder repeatedly lets that state interact with the input features, outputting the translated words one at a time.
Multi-object tracking can be seen as a similar problem, where the output is a set of sequences and each sequence is one object's track. If each track also had a hidden state that interacts with image features frame by frame and outputs a box frame by frame, the ID would naturally follow that hidden state, and no further matching would be needed.
MOTR extends DETR's object queries into track queries, which serve as the tracks' hidden states. Doing this means solving two problems first:
- How does a query keep following the same object? DETR's object queries can be assigned to any object in the scene in any frame; they have no notion of identity.
- What about objects that appear and leave? The number of objects in the scene keeps changing, so the number of track queries has to change with it.
Solving the problem
Two kinds of queries

MOTR splits queries into two kinds. Let's walk through this figure:
- Detect queries: DETR's original fixed-size set of learnable object queries, the same in every frame. They're responsible only for finding newly appeared objects.
- Track queries: a variable-size set of queries, each standing for a track that's already being followed.
T1: At the start there are no tracks yet, so the set of track queries is empty. Only detect queries go into the decoder, and two of them find object 1 and object 2.
T2: The hidden states of the two queries that found objects 1 and 2, after passing through the decoder, become track queries for the next frame. At T2, the decoder's input is "two track queries + a set of detect queries." The two track queries each output a new box for object 1 or 2, with the same ID; one of the detect queries finds the newly appeared object 3.
T3: Object 3's hidden state also joins the track queries, so now there are three.
T4: Object 2 has left, and its track query is removed from the set.
Throughout the whole process, each track query stands for one object only, from creation to removal. The tracking result is simply the boxes output by the same track query in every frame, joined together.
How labels are assigned
During training, DETR uses bipartite matching to decide which ground-truth object each query is responsible for. MOTR changes this into Tracklet-Aware Label Assignment (TALA), which handles the two kinds of queries separately:
- Detect queries: bipartite matching only against the objects that newly appear in this frame.
- Track queries: no matching; they keep the assignment from the previous frame. The query responsible for object 1 in the previous frame is still responsible for object 1 in this one.
Written as a formula, the assignment of track queries in frame , , is the previous frame's track-query assignment plus the previous frame's assignment for new objects:
At inference time there are no labels, so how do detect queries know not to go after objects that are already being tracked? Both kinds of queries go into the decoder together, and self-attention suppresses the scores of detect queries that find already-tracked objects. It's the same reason DETR doesn't output duplicate boxes even without NMS.
Overall architecture

The backbone is ResNet-50 and the detection framework is Deformable DETR. Each frame passes through the backbone and the Transformer encoder to get features. The decoder's input is the previous frame's track queries concatenated with the detect queries. Its output hidden states are used to predict boxes and classes, and are also sent into the Query Interaction Module (QIM), which produces the next frame's track queries.
QIM is responsible for two things.
Objects entering and leaving. During training, if an object disappears from the ground-truth annotation, or its predicted box has an IoU below 0.5 with the ground-truth box, its hidden state is discarded; new objects are kept according to the detect queries' matching results. At inference time, classification scores are used instead. A detect query becomes a new track only if its score is above the entrance threshold , and a track query is removed only if its score stays below the exit threshold for consecutive frames.
The Temporal Aggregation Network (TAN). This is a modified Transformer decoder layer. For each tracked object, the previous frame's track query is added to this frame's hidden state and used as the query and key of multi-head self-attention, with this frame's hidden state as the value, followed by a feed-forward network. This gives track queries a shortcut for bringing in historical information. TAN's output is then concatenated with the hidden states of new objects to form the next frame's track queries.
Computing the loss over a whole clip
The Kalman filter's constant-velocity assumption is written by people; MOTR's motion model is meant to be learned from data. But trained on only two adjacent frames, the network sees only very short motion and can't learn long-range changes such as an object reappearing after occlusion.
The authors propose the Collective Average Loss (CAL). During training, a video clip is fed in at once, the predictions from every frame are collected, and the loss over the whole clip is divided by the total number of ground-truth objects in the clip:
is the number of ground-truth objects in frame . The single-frame loss is the same as DETR's, a weighted sum of focal loss, L1 loss and GIoU loss.
The clip length starts at 2 frames and is gradually increased to 3, 4 and 5 frames over the course of training. The authors say this progressive increase improves training efficiency and stability.
A few training tricks
In real videos, new objects appearing and old ones leaving are both relatively rare. The authors use two random operations to create more examples:
- With probability , some track queries are randomly deleted, giving detect queries a chance to practice "finding new objects."
- With probability , false-positive queries are inserted into the track queries, so the model practices "letting objects that don't exist exit."
In addition, the frames of a training clip aren't taken consecutively; they're sampled at random intervals, simulating different frame rates and larger motion.
Compared with their contemporaries TransTrack and TrackFormer, the authors stress that MOTR needs no post-processing at all. TransTrack detects pairs of short tracklets in two adjacent frames and then links them into long tracks with IoU matching. TrackFormer also has the idea of track queries, but learns only between two adjacent frames and still needs track NMS and re-ID features to filter out duplicate tracks. The authors also state explicitly that this work started independently before TrackFormer and TransTrack were posted to arXiv.
Discussion
MOTR is trained on 8 V100s, with each batch being a 5-frame clip. On MOT17 it's trained for 200 epochs, on the MOT17 training set plus about 5000 static images from the CrowdHuman validation set (with pseudo-clips generated by CenterTrack's random shifts). Input images are resized so the short side is 800, and inference runs at about 7.5 FPS on a V100.
MOT17: an awkward result
On the MOT17 test set, MOTR scores HOTA 57.8, IDF1 68.6, MOTA 73.4 and 2439 ID switches.
Compared with the other Transformer methods, its IDF1 is 4.7 higher than both TransTrack and TrackFormer; its HOTA is 3.7 higher than TransTrack, and its MOTA is much higher than TrackFormer's 65.0. But TransTrack's MOTA (74.5) is higher than MOTR's. The authors guess that TransTrack splits detection and tracking into two branches and detects better, whereas in MOTR, detect queries are suppressed by track queries, which limits the detection of new objects.
Compared with CNN methods, it's even more awkward. ByteTrack has HOTA 63.1, MOTA 80.3 and IDF1 77.3, and MOTR loses on every one. The authors describe it in the paper, in so many words, as "frustratingly inferior." Their explanation is that results on MOT17 are determined mainly by detection quality, and since everyone uses a different detector, it's hard to compare motion-modeling ability fairly, so MOT17 isn't enough to fully evaluate MOTR.
DanceTrack: home turf

DanceTrack is a dataset of dance scenes. The dancers dress alike, move in complex ways and cross each other constantly. Detection isn't hard; association is.
Here MOTR's HOTA is 54.2, 6.5 higher than ByteTrack's 47.7; its AssA is 40.2 against 32.1, 8.1 higher. But its DetA of 73.5 is below CenterTrack's 78.1 and TransTrack's 75.9, and its MOTA of 79.7 is the lowest in the whole table.
This table lays MOTR's strengths and weaknesses side by side very clearly. High AssA shows that track queries really have learned to keep identities across frames; low DetA and MOTA show that it isn't as good at finding objects as the other methods. That's exactly what the previous note on HOTA said: when you look at HOTA, glance at DetA and AssA too.
On the multi-class BDD100K validation set, MOTR's mMOTA is 32.0, with 3493 ID switches, far fewer than the baseline's 8315. The authors argue that re-ID-based methods struggle when there are many classes and many objects, whereas MOTR can handle multi-class tracking just by changing the number of classes in the classification head.
Ablations: what each design is worth

The ablations are trained on the MOT17 training set and validated on the MOT15 training set.
The left table starts from the most basic version. With only object queries and no track queries, objects in every frame are treated as new, ID switches reach 33198 and IDF1 is just 1.2. Adding track queries brings IDF1 up to 49.8 and ID switches down to 562. Track queries alone are enough to make tracking work.
TAN and CAL each help on their own. Adding only TAN raises MOTA by 7.8 and IDF1 by 13.6; adding only CAL raises MOTA by 10.4. With both, MOTA is 53.2, IDF1 70.5 and ID switches 155.
The right table covers CAL's clip length. Going from 2 frames to 5, MOTA rises from 44.9 to 53.2, IDF1 from 63.4 to 70.5, and ID switches fall from 257 to 155. The path isn't smooth: at length 3, IDF1 even drops to 59.4 and ID switches rise to 424. But the longest, 5 frames, is best on all three metrics. The authors observe that multi-frame CAL clearly reduces duplicate boxes, ID switches and missed detections in occlusion scenes.
There are also ablations of the training tricks. The probability of randomly deleting track queries works best at 0.1. The probability of inserting false-positive queries gives the highest MOTA at 0.3, but IDF1 declines steadily from 0.1 onward. For the random sampling interval during training, 10 works best, with 155 ID switches; intervals of 3 and 5 give 218 and 324, and an interval of 12 is slightly worse again. The authors think intervals that are too small make it easy to get stuck in a local optimum, while with intervals that are too large, the model can't learn changes over such long distances.
Limitations the authors admit
The authors list two limitations at the end of the paper. First, the ability to detect new objects is far from ideal, and MOTA isn't good enough. Detect queries are suppressed so they don't go after tracked objects, which may go against the nature of object queries and also limits the detection of new objects. Second, queries are passed from frame to frame sequentially, which limits training efficiency; they tried parallel decoding like VisTR's, and it couldn't handle the complexity of multi-object tracking.
Afterward
The first limitation got an answer quickly. A little over a year later, the same team published MOTRv2, which feeds detections from a pretrained YOLOX into MOTR as anchor-style queries, handing detection to a dedicated detector and letting MOTR focus on association. The conflict between end-to-end tracking and detection later became a recurring topic in this line of work, and follow-ups such as MeMOTR and MOTIP all compare themselves against the MOTR family.
ByteTrack confirmed MOTR's position on MOT17 from the other direction. On the same MOT17, a good detector plus a carefully designed set of matching rules gets scores that MOTR can't reach.
Conclusion
Extend DETR's object queries into track queries that follow objects, assign labels with TALA, handle entries, exits and temporal aggregation with QIM, and train with a collective average loss over a whole clip. MOTR needs no hand-written matching rules from start to finish, and on DanceTrack, where association is hard, its HOTA is 6.5 higher than ByteTrack's.
The price shows up in the DetA and MOTA columns. With track queries and detect queries crowded into the same decoder, new objects are harder to find; on MOT17, where detection dominates the score, it loses to the much simpler ByteTrack.
As for that line in the paper, "frustratingly inferior," it's probably the most honest sentence in this whole series of papers.