CO-MOT: Boosting End-to-end Transformer-based Multi-Object Tracking via Coopetition Label Assignment and Shadow Sets
- Publication
- ICLR 2025Conference
- First submitted to arXiv
- arXiv:2305.12724
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingcomot2023,
title = {{CO-MOT: Boosting End-to-end Transformer-based Multi-Object Tracking via Coopetition Label Assignment and Shadow Sets}},
author = {Feng Yan and Weixin Luo and Yujie Zhong and Yiyang Gan and Lin Ma},
year = {2023},
eprint = {2305.12724},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.12724}
}
MOTR hands tracking over entirely to a Transformer. Each person gets one query, which follows them from frame to frame, with no Kalman filter and no Hungarian algorithm.
It's an elegant design, but its results never caught up with "detect first, then match" methods such as ByteTrack. The later MOTRv2 simply bolted on a YOLOX detector. The results went up, but so did the computation, and most of the original "end-to-end" advantage was gone.
Feng Yan, Weixin Luo and colleagues at Meituan went back to examine MOTR and found that the problem lies not in tracking but in how labels are assigned during training.
Defining the problem
Detection queries left on the sidelines
MOTR's decoder has two kinds of queries. Track queries handle people already being tracked and are carried over from the previous frame each time; detection queries look for people who have just appeared.
Label assignment during training is called TALA (tracking aware label assignment). It first assigns people already being tracked to their corresponding track queries, and only the leftover newborn objects go into bipartite matching with the detection queries.
The problem is that there are too few "leftovers." The authors counted on DanceTrack that the ratio of newly appearing people to people already being tracked is 213 to 25,483. The dancers on stage are almost all there from the first frame, and few enter afterward. Most of the time, detection queries have no positives to learn from.
Worse, TALA amounts to teaching detection queries "hands off any object someone is already tracking." At inference, once a track query loses its person, the detection queries won't pick that person up either.

The top row is MOTR's tracking result, and the bottom row is the same model with its track queries removed, doing detection only. Everyone in the bottom row gets a box. So the issue isn't that MOTR can't detect; it's that when track queries are around, detection gets suppressed.
Scoring every box as a detection
The authors take every box the decoder outputs, tracking or detection alike, treat them all as detections, and measure them with mAP:
| Model | Track queries in training | Track queries at inference | mAP |
|---|---|---|---|
| MOTR | ✓ | ✓ | 42.5 |
| MOTR | ✓ | 60.6 | |
| MOTRv2 | ✓ | ✓ | 70.7 |
| MOTRv2 | ✓ | 72.1 | |
| CO-MOT | ✓ | ✓ | 69.1 |
Compiled from Table 1 of the paper, DanceTrack validation set, last decoder layer.
For the same MOTR, removing the track queries at inference makes mAP jump from 42.5 to 60.6, a gain of 18.1. MOTRv2, with detections supplied by YOLOX, gains only 1.4 when its track queries are removed; the bolted-on detector has all but solved the detection problem.
CO-MOT's goal is to bring back MOTR's own detection ability without relying on a bolted-on detector.
Solving the problem

CO-MOT's architecture is identical to MOTR's. The only change is to label assignment during training, and it comes in two parts.
Cooperating and competing at once
The first change is called COLA (coopetition label assignment); coopetition is a blend of cooperation and competition.
The decoder has six layers, and COLA lets the detection queries in the first five be matched not only to new objects but also to people already being tracked. In other words, in the middle layers the same person may have both a track query and a detection query predicting them.
The last layer still uses TALA, where people already being tracked belong only to track queries. Otherwise the same person would sprout two tracks at inference.
The benefit comes from self-attention in the decoder. All the queries look at one another; the detection queries see this frame's image, while the track queries carry memories of the past. When both are predicting the same person, the detection query can pass this frame's information to the track query, giving it a hand when appearance changes a lot.
They cooperate in the middle layers and compete in the last one, hence the name coopetition.
Every query has shadows
The second change is the shadow set.
DETR-style models use one-to-one matching, so only one query gets a positive for each object. Co-DETR and H-DETR respond with one-to-many matching, having several queries learn the same object. CO-MOT instead copies each query into shadows and makes the whole set jointly responsible for the same object.
How the shadows are initialized matters a great deal. Fully random, and the shadows in a set differ too much to converge easily; fully identical, and there's no point in having a set. The authors initialize a set to the same learnable vector plus noise with a standard deviation of . The differences are barely visible at first and grow to around by the end of training.
During training, each set of shadows picks one representative to take part in matching. A set of track queries is matched directly to its target from the previous frame; for a set of detection queries, the matching cost of each shadow against each object is computed first, and the one with the highest cost becomes the representative. In other words, who a set is matched to is decided by its worst member. The other shadows in the set share the matching result and receive positives together.
At inference it's the other way round. The lowest confidence score in the set decides whether the set has caught someone; if it passes, the box of the highest-scoring shadow is output.
Counting positives: an example
Let's take one training clip and see how much more the detection queries get to learn from these two changes.
CO-MOT trains on 5 frames at a time. Suppose 10 dancers are all in the picture from frame 1, nobody enters or leaves partway through, and the decoder has 6 layers.
- TALA. Frame 1 has no track queries yet, so all 10 people are new objects and go to detection queries. In frames 2 to 5, everyone goes to track queries. Detection queries get only 10 positives per layer, 60 across the 6 layers.
- COLA. In the first 5 layers, detection queries can be matched to every object in all 5 frames, 50 per layer, 250 in total; the last layer stays at 10. That adds up to 260.
- Add shadows. All 3 shadows in each set get the positives, so the detection queries' positives become 780.
Worked example for this note.
The track queries don't lose a single positive, while the detection queries go from 60 positives to 780.
Discussion
What each change contributes
| Setup | HOTA | DetA | AssA | IDF1 |
|---|---|---|---|---|
| MOTR baseline | 56.4 | 71.8 | 44.6 | 57.5 |
| + COLA | 60.2 | 73.2 | 49.7 | 62.4 |
| + shadow sets | 59.0 | 72.6 | 48.2 | 59.6 |
| Both | 61.8 | 73.5 | 52.2 | 63.3 |
Compiled from Table 4 of the paper, DanceTrack validation set.
COLA alone adds 3.8 HOTA, shadow sets alone add 2.6, and the two together add 5.4.
Interestingly, DetA only goes from 71.8 to 73.5, while AssA jumps from 44.6 to 52.2. On the surface, COLA gives detection queries more positives and looks as if it should help detection; what it actually helps is association.
Are detection queries really helping?

To check this, the authors plot the attention weights at each decoder layer. Two boxes with an IoU above 0.7 are treated as predicting the same person.
In CO-MOT, the detection query predicting the same person contributes more than 15% to the track query from layer 3 onward, and at layers 4 and 6 it even exceeds the track query's attention to itself. In MOTR, the same value at layer 4 is only 3.74%, against 30.74% for CO-MOT.
In MOTR, detection queries and track queries are responsible for different people and have little to say to each other. CO-MOT has them watch the same person in the middle layers, and the information flows across.
Shadow details
- Initialization. The same vector plus tiny noise gives a HOTA of 61.5, an exact copy 60.6, and fully random 59.6.
- Choosing the representative. Maximum cost in training with lowest score at inference is the best combination; switching inference to the highest score is the worst, whatever the training choice. Letting the hardest shadow decide the matching during training forces the model to learn every shadow well.
- Number of shadows. 3 is best (61.8), and 2 gives 61.5; beyond that, convergence gets hard.
- How many layers use COLA. First layer only 59.0, first three 59.6, first five 59.9; the more layers, the better.
Compared with other methods
On the DanceTrack test set, CO-MOT trained only on DanceTrack reaches a HOTA of 65.3, 11.1 above MOTR and also above MeMOTR (63.4), which likewise uses ResNet-50. CO-MOT+, trained with CrowdHuman added, reaches a HOTA of 69.4, only 0.5 behind MOTRv2 (69.9) with its bolted-on YOLOX, and almost the same AssA (58.9 vs. 59.0).
| Model | Params | FLOPs | Speed | HOTA |
|---|---|---|---|---|
| MOTR | 40M | 173G | 19 FPS | 54.2 |
| MOTRv2 | 139M | 455G | 14 FPS | 69.9 |
| CO-MOT+ | 40M | 173G | 19 FPS | 69.4 |
Compiled from Figure 4 of the paper, DanceTrack test set.
This table is CO-MOT's most convincing argument. With the same parameter count and computation as MOTR, it catches up with MOTRv2 using only 38% of its computation, and runs 1.4 times as fast. Only the training changes; inference costs nothing extra.
On BDD100K, its TETA is 52.8, above MOTR's 50.7 but below MOTRv2's 54.9; its AssocA, however, is 56.2, above MOTRv2's 51.9.
Still stuck on MOT17
On the MOT17 test set, CO-MOT's HOTA is 60.1, better than MOTR's 57.8 but still below ByteTrack's 63.1; its MOTA of 72.6 trails ByteTrack's 80.3 by an even wider margin.
The authors offer two explanations. First, MOT17's training set has only about 5,000 frames, too few for a Transformer. Second, MOT17 has many small targets. Boxes smaller than 0.5% of the image area make up 60.26% of MOT17 but only 1.17% of DanceTrack. The failure cases the paper shows are mostly clearly visible people who weren't detected.
CO-MOT frees MOTR's detection ability from being suppressed by tracking, but it can't change how well MOTR itself detects small objects.
Conclusion
CO-MOT finds a blind spot in how MOTR is trained. Detection queries are taught "don't touch anyone already being tracked," so when a track query loses someone, nobody can pick them back up.
COLA has both kinds of queries look at the same person together in the middle layers, and shadow sets give each object several queries learning it together. Both changes affect only training; the inference architecture is exactly MOTR's.
On DanceTrack, CO-MOT+, with no bolted-on detector, is only 0.5 HOTA behind MOTRv2 and its YOLOX, at less than 40% of the computation. On MOT17, with its many small targets and scarce data, the gap to detect-then-match methods is still there.