Multiple Object Tracking as ID Prediction
- Publication
- CVPR 2025Conference
- First submitted to arXiv
- arXiv:2403.16848
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingmotip2024,
title = {{Multiple Object Tracking as ID Prediction}},
author = {Ruopeng Gao and Ji Qi and Limin Wang},
year = {2024},
eprint = {2403.16848},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2403.16848}
}
By this point, we've seen two broad families of trackers.
One is detect-then-associate. Detection is left to a dedicated detector, and association relies on hand-designed rules: a Kalman filter predicts positions, IoU or appearance distance gives the cost, the Hungarian algorithm does the matching, plus a pile of thresholds. These rules work very well on pedestrians, but every new situation needs someone to analyze it, change the rules and retune the parameters. From SORT to Deep OC-SORT, and on to C-BIoU and UCMCTrack, they all take this road.
The other is the end-to-end MOTR family, which passes a track query along from frame to frame and does detection and tracking together inside the decoder. Association no longer relies on hand-written rules, but the two kinds of queries crammed into one decoder interfere with each other, and detection often suffers for it. MOTRv2 simply hands detection back to YOLOX.
Ruopeng Gao and Limin Wang of Nanjing University, together with Ji Qi of China Mobile (Suzhou) Software Technology, wanted to know whether the detect-then-associate division of labor could be kept while association is handed over entirely to a model to learn.
Their answer is MOTIP. It's Ruopeng Gao's next paper after MeMOTR, and the method is surprisingly simple.
Defining the problem
Association looks a lot like a classification problem. In the training data, every track has an ID. Given a detection box, deciding which track it belongs to is just predicting its ID, isn't it?
The problem is inference. Everyone who appears in a test video is someone the model has never seen, with new IDs too, and a model can't classify a person into a class that never appeared during training.
So methods like FairMOT learn appearance features with a classification loss during training, but have to switch approach at inference, taking the features out to compute cosine similarity and then matching them with hand-written rules. Training and inference are really doing two different things.
The authors think the sticking point is that tracking IDs are fundamentally different from ordinary classification labels.

In ordinary classification, a "cat" label is a cat; the label carries meaning. Tracking IDs don't. The four people written as 1, 2, 3, 4 in the ground-truth file could just as well be 8, 5, 7, 3, as long as each person keeps the same number from start to finish. An ID is just a number tag saying "these boxes are the same person"; what the number actually is doesn't matter.
Solving the problem
In-context ID prediction
Since the number itself doesn't matter, let the model look at "the number tags hanging on the historical tracks" and predict which one each new detection should take.
Concretely, each past track is randomly given a number , with . When a new frame comes in, if a detection belongs to the -th track, the correct answer is .
The numbers are hints handed to the tracks, so the authors call this in-context ID prediction. However many new faces turn up at test time, what's predicted is always a number between 1 and , all within the range seen in training. The numbers can be handed out any way you like, and a prediction counts as correct as long as it agrees with the history; that is exactly what the three rows of Figure 1 show.
Three parts

The whole architecture has just three parts.
DETR detector. Deformable DETR with a ResNet-50 backbone. Its advantage is that the decoder's output embeddings can serve directly as object features, with no need to crop boxes or do RoI pooling separately. Each object gets a 256-dimensional feature .
ID dictionary. A set of learnable 256-dimensional embeddings:
The first stand for ordinary numbers, and the last one, , stands for "new object." The authors don't use one-hot encoding, because it's bad for training and would tie the maximum number to the vector dimension. The default is ; it just has to be no smaller than the largest number of objects in a single frame.
ID decoder. A standard Transformer decoder with self-attention and cross-attention layers alternating, followed by a linear classification head that outputs probabilities over classes.
How an object becomes a token
Let's walk along a single track.
A track that's already being followed has been given the number . Each of its boxes over the past frames has a DETR feature . The feature and the number's embedding are simply concatenated:
Each token is 512-dimensional. The first half is "what it looks like," the second half is "which number it's wearing." All of this track's tokens over the past frames form its history. is 29 frames on DanceTrack and 59 on SportsMOT.
Nobody knows yet who the detections in the current frame are, so they all get the new-object token:
Next, the current detections serve as queries and all the historical tokens as keys and values, and they go into the ID decoder. Cross-attention in the decoder lets each detection search the history for "which number the most similar-looking string of tokens is wearing"; self-attention lets the detections in the same frame exchange information, so two people who look alike don't grab the same number. Finally the classification head outputs the probability of each detection belonging to each number.
If the detection belongs to the -th track, the answer is ; if it's someone who has just appeared, the answer is the new-object class. Association thus becomes a classification problem with classes, which can be trained directly with cross-entropy, added to DETR's detection losses:
The four weights are 2, 5, 2 and 1.
Inference: handing out and taking back the tags
At inference the procedure is straightforward:
- Numbers from 1 to are handed out to tracks in order, with no duplicates. When a track ends, its number is taken back for new tracks to use. So however long the video and however many people appear, all that matters is that no more than tracks are being followed at the same time.
- DETR's outputs are first filtered at a confidence of 0.3.
- The remaining detections go into the ID decoder, and each picks the number with the highest probability, provided it's above 0.2.
- If two detections in the same frame pick the same number, only the more confident one keeps it.
- Detections that didn't get a number and have a detection confidence above 0.6 are treated as new objects and given a new number tag.
By default there's no Hungarian algorithm. The authors think a more elaborate assignment strategy might add a little more, but what they want to verify is the capability of the model itself.
The authors also counted that in most cases, fewer than 40% of the numbers are in use. If you really do hit an extremely crowded scene, just raise , much as DETR uses 300 queries by default.
In training, make mistakes on purpose
During training, the historical tracks are built from ground truth, and every one of them is spotless. Inference is different. People get occluded, the model hands out wrong numbers, and the mistakes get written into the history that follows.

To get the model used to imperfect histories, the authors add two kinds of trajectory augmentation:
- Random occlusion: with probability , a stretch of tokens in the middle of a track is removed, simulating someone being blocked.
- Random switching: with probability , the numbers of two tracks in the same frame are swapped, simulating the model mixing up two similar people.
Both are set to 0.5.
Also, because detection and association are two separate modules, training can be parallelized. Each step takes a clip of frames. DETR can run the forward pass on every frame at once, while the ID decoder uses an attention mask to make sure it can't see the future; DETR only needs to backpropagate through 4 of those frames. With 8 RTX 4090s, the authors train on DanceTrack in under a day.
Discussion
The authors deliberately keep the comparison conditions consistent. The main results all use Deformable DETR with ResNet-50, and no extra detection datasets. In their view, training on fake videos made by shifting static images such as CrowdHuman is bad for end-to-end methods, especially ones that need long-term modeling.
DanceTrack
| Method | HOTA | DetA | AssA | IDF1 |
|---|---|---|---|---|
| OC-SORT | 55.1 | 80.3 | 38.3 | 54.6 |
| C-BIoU | 60.6 | 81.3 | 45.4 | 61.6 |
| DiffMOT | 62.3 | 82.5 | 47.2 | 63.0 |
| MeMOTR | 63.4 | 77.0 | 52.3 | 65.5 |
| CO-MOT | 65.3 | 80.1 | 53.5 | 66.5 |
| MOTIP | 69.6 | 80.4 | 60.4 | 74.7 |
| MOTRv2 (extra data) | 69.9 | 83.0 | 59.0 | 71.7 |
| MOTIP (extra data) | 72.0 | 81.8 | 63.5 | 76.8 |
Compiled from Table 1 of the paper, DanceTrack test set.
Without extra data, MOTIP reaches 69.6, 4.3 points above CO-MOT, which also uses Deformable DETR, with AssA 6.9 points higher and IDF1 more than 8 points higher. The version with extra data reaches 72.0, ahead of MOTRv2.
MeMOTR's 63.4 in the table is its version switched to standard Deformable DETR; the main table of the MeMOTR paper reports 68.5 with DAB-Deformable-DETR. Even against 68.5, MOTIP is still a notch higher, but the two use different detection frameworks.
SportsMOT and flocks of birds
On SportsMOT, using only the training set, MOTIP's HOTA is 72.6 and its AssA 63.2, higher than OC-SORT under the same conditions (68.1) and the Deformable DETR version of MeMOTR (68.8), and also higher than DiffMOT (72.1), which uses extra data.
Still, the best performer on SportsMOT is the Deep-EIoU school of approach. It trains a YOLOX detector on the training set plus the validation set, adds a ReID model, and gets a HOTA of 77.2. The two differ in both training data and detector, so they can't be compared directly, but it's also a reminder that in sports scenes, a strong detector plus simple rules is still very competitive.
The authors also tested on BFT, a bird-tracking dataset with 22 species and 106 videos. Birds fly in three-dimensional space, their movements are more irregular than those of people on the ground, and they have no clothes to tell them apart. MOTIP's HOTA is 70.5, OC-SORT's 66.8 and ByteTrack's 62.5.
Self-attention and the Hungarian algorithm
The ablations are run on the DanceTrack validation set:
| Self-attn | Hungarian | Traj. aug. | HOTA | AssA |
|---|---|---|---|---|
| 57.7 | 43.9 | |||
| ✓ | 59.7 | 47.2 | ||
| ✓ | 59.5 | 47.2 | ||
| ✓ | ✓ | 59.9 | 47.9 | |
| ✓ | ✓ | 62.2 | 51.5 | |
| ✓ | ✓ | ✓ | 62.2 | 51.8 |
Compiled from Table 4 of the paper, DanceTrack validation set.
- Without self-attention, the Hungarian algorithm helps. HOTA goes from 57.7 to 59.7. Without self-attention, each detection picks a number on its own, unaware of what the others picked, so a global assignment is needed to avoid conflicts.
- With self-attention, the Hungarian algorithm is almost useless. It adds only 0.4 points, and once trajectory augmentation is in, the score is exactly the same. The authors' explanation is that self-attention lets the detections in the same frame exchange identity information while decoding, so the model learns to do the global assignment by itself.
- Trajectory augmentation adds 2.7 points. From 59.5 to 62.2, the biggest step in this table.
The augmentation rates need tuning too. With random occlusion alone, 0.5 is best (60.7), and occluding at 1.0 actually drops it to 58.6; adding random switching on top of 0.5 occlusion, 0.5 is best (62.2), and 0.8 falls to 59.8.
Compared with ReID
MOTIP also tracks with object features, so it looks a lot like a ReID method. The authors compare it with two common ways of training ReID, FairMOT's classification style and a contrastive learning style, both of which use cosine similarity at inference.
With DETR frozen and only the association part trained, the FairMOT style gets a HOTA of just 29.4, the contrastive style 41.0, and MOTIP's ID prediction 55.4. When all parameters are trained together, the ReID methods, even with the Hungarian algorithm plus a trajectory enhancement module built the same way as the ID decoder, reach at best 54.5; MOTIP gets 59.5.
With the same features, switching to training that "predicts the number from the context" makes a difference of at least 5 points. The authors argue that ReID methods integrate historical information with hand-written rules and compute similarities independently, which limits the model's flexibility; MOTIP lets the model decide for itself how to look at the historical tracks.
Limitations
On MOT17, MOTIP's HOTA is 59.3, about the same as CO-MOT (60.1) and MOTRv3 (60.2) among end-to-end methods, but clearly behind heuristic methods such as ByteTrack (63.1). The authors' explanation is that MOT17 has only 7 training videos, too few for a method that has to learn from data, while heuristic methods have been polished for pedestrians' linear motion for nearly a decade.
As for speed, on a single RTX A5000, MOTIP runs at 12.7 frames per second in FP32, about the same as MOTR's 13.1; switching to FP16 gets it to 22.8.
The authors also explain that the design is deliberately kept simple, to verify that the idea of "tracking as ID prediction" works, and that there is plenty of room to add other cues such as motion and depth.
Conclusion
MOTIP rewrites association in multi-object tracking as an in-context ID prediction problem. Numbers are handed out randomly to the historical tracks, and the model only has to look at these numbers and predict which one each new detection should take. The whole model is just DETR, an ID dictionary and a standard Transformer decoder, with no Kalman filter, no cost matrix and not even a Hungarian algorithm, yet it beats earlier methods on DanceTrack, SportsMOT and bird tracking.
The most interesting thing in the ablations is the Hungarian algorithm going redundant. A component that has been there ever since SORT is replaced in MOTIP by a layer of self-attention.
Looking back, the evaluation metrics had been treating IDs this way all along. CLEAR MOT builds correspondences frame by frame, and IDF1 matches true identities to computed identities one-to-one over the whole video. Neither cares whether the tracker hands out number 3 or number 87, only whether the same person keeps holding the same number tag. MOTIP just has the model understand IDs the same way.