Object-Centric Multiple Object Tracking
- Publication
- ICCV 2023Conference
- First submitted to arXiv
- arXiv:2309.00233
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingocmot2023,
title = {{Object-Centric Multiple Object Tracking}},
author = {Zixu Zhao and Jiaze Wang and Max Horn and Yizhuo Ding and Tong He and Zechen Bai and Dominik Zietlow and Carl-Johann Simon-Gabriel and Bing Shuai and Zhuowen Tu and Thomas Brox and Bernt Schiele and Yanwei Fu and Francesco Locatello and Zheng Zhang and Tianjun Xiao},
year = {2023},
eprint = {2309.00233},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2309.00233}
}
In 1989, cognitive scientist Zenon Pylyshyn proposed a theory of visual-spatial perception. In Pylyshyn's view, the human visual system has a small set of "indexes," each of which sticks to one object; when the object moves, brushes past other objects, or is even partly blocked, the index stays stuck to it.
That sounds a lot like what multi-object tracking does, and a lot like the slots in Slot Attention.
Zixu Zhao, Tianjun Xiao and colleagues at Amazon AWS, joined by Francesco Locatello, co-first author of Slot Attention, wanted to see whether object-centric learning could be used directly for tracking. If it could, tracking would no longer need so many annotated boxes, let alone labels saying which identity each box belongs to.
Defining the problem
Tracking annotation is expensive
Mainstream trackers detect first, then associate. The detector needs a large number of boxes to train on; if the association module is to learn appearance features, it also needs to know which boxes are the same person, that is, ID annotations.
ID annotations cost more than boxes. An annotator has to watch the whole video and confirm that this person in frame 37 is the same as that person in frame 512.
Object-centric learning takes another path. Methods like Slot Attention need no annotation at all; just by reconstructing the image, they split a scene into slots, each roughly corresponding to one object. SAVi extends this to video, letting slots carry over across frames.
Slots that split, slots that swap
But using SAVi directly for tracking runs into two problems.

- Splitting. The same object is caught piecemeal by several slots, or one slot catches only part of an object. This is called the part-whole problem.
- Swapping. The same slot catches the green cube in frame 1 and something else by frame 12. For tracking, that's an ID switch.
Object-centric learning is usually evaluated with FG-ARI, a metric that only looks at how good the grouping is in each frame; slots swapping "ownership" with one another isn't heavily penalized. So these two problems went unaddressed for a long time.
Solving the problem

OC-MOT uses the object-centric model as its detector, frozen, and attaches a memory module behind it. Only the memory module is trained.
Memory: each buffer is a track
The memory consists of buffers. Each buffer stores the representations of one object over the past few frames, first in, first out, keeping at most frames. Seen another way, each buffer is a track.
Before each frame begins, a small GPT-2 with only 1.6 million parameters reads the history of every buffer and predicts what each object should look like in this frame. This step is called rollout.
Rollout combines how an object looked over several past frames. If an object is half blocked in this frame, rollout still predicts the whole of it.
Indexing: each slot finds its seat
Next, each slot has to find which buffer it belongs to.
The authors use a multi-head attention with the slots as queries and the rollouts as keys and values, and take its attention weights to get an index matrix . Row gives the probability of slot being assigned to each buffer.
Duplicate slots, and slots that catch only part of an object, may all point to the same buffer, and fragments belonging to the same object should go back to the same seat anyway.
Merging: putting the fragments back together

The second multi-head attention runs the other way, with the rollouts as queries and the slots as keys and values, and uses the index matrix as a mask: each buffer can only see the few slots assigned to it.
The weights renormalized after masking are the merging proportions. In the paper's example, two of three slots point to the same buffer, and the merging weights might be . The second slot was assigned here, but it doesn't actually belong to this object; its weight is very low, so it's effectively filtered out.
The merged result is written back to memory and becomes this frame's tracking output.
Training without ID labels
The training goal is to have each slot assigned to the buffer that "looks most like it." Likeness is judged by decoding both with the frozen decoder into masks and images and comparing them: cross-entropy on the masks compares shape, squared error on the object reconstructions compares color, and the distance between representation vectors compares features.
The hard part is the indexing. In the end each slot can be assigned to only one buffer. The intuitive approach is to take the argmax, but argmax isn't differentiable; force it through with the straight-through trick, and the model only sees the one it picked, so it easily gets stuck in a wrong assignment.
The authors borrow the idea of the EM algorithm, treating the index matrix as probabilities and minimizing the expected cost:
An example shows the difference. Take one slot and two buffers, with matching costs of 0.2 and 1.0 against them. The current index, , favors the wrong buffer.
| Approach | Loss | What it sees |
|---|---|---|
| Take argmax | 1.0 | Buffer 2 only |
| Expected cost | 0.4 × 0.2 + 0.6 × 1.0 = 0.68 | Both |
Worked example for this note.
Under the expected cost, move 0.1 of probability from buffer 2 to buffer 1 and the loss becomes 0.60, down 0.08, which is exactly . The gradient naturally pushes probability toward the cheaper side. The whole process only compares how alike slots and memory are, without using a single ID label.
Only at inference is the index matrix binarized, so that each slot is assigned to only one buffer. In the first frame, the buffers are initialized from the slots, after removing duplicate slots whose masks overlap each other too much; after that, a slot that overlaps none of the rollouts opens a new buffer. A buffer is closed only once its object has been gone for more than 5 frames.
Discussion
Two datasets
The experiments use two synthetic datasets. CATER is a collection of 3D geometric objects moving around on a table and occluding one another; FISHBOWL, recorded from a WebGL aquarium, has 20,000 training videos plus 1,000 validation and test videos of 128 frames each; there the schools of fish occlude each other more heavily and the background is more complex.
On CATER, the object-centric model is SAVi with no annotation at all. FISHBOWL is harder, so the authors add a little supervision to DINOSAUR: in each 128-frame video, masks are annotated on 8 random frames, that is, 6.25%. Neither dataset gets ID annotations.
| Method | Det. labels | ID labels | IDF1 | MOTA | ID sw. |
|---|---|---|---|---|---|
| SAVi | 6.25% | — | 46.9 | 32.3 | 12,504 |
| SORT | 6.25% | — | 68.4 | 64.3 | 15,278 |
| IOU | 6.25% | — | 71.3 | 66.6 | 10,306 |
| Visual-Spatial | 6.25% | — | 74.6 | 68.1 | 8,754 |
| OC-MOT | 6.25% | — | 77.9 | 70.3 | 5,898 |
| MOTR | 100% | 100% | 81.6 | 79.8 | 4,185 |
Compiled from Table 1 of the paper, FISHBOWL.
For fairness, SORT, IOU and Visual-Spatial take the output of the same object-centric model as their detections. OC-MOT has the fewest ID switches of the unsupervised methods, 60% fewer than SORT. Fish don't swim in straight lines, and neither the Kalman filter nor IoU can really keep up.
Against the fully annotated MOTR, it is still 3.7 behind on IDF1 and 9.5 behind on MOTA. The authors attribute the gap mainly to detection: DINOSAUR predicts masks at a very low resolution, so the boxes aren't accurate enough.
On CATER the gap is smaller. OC-MOT, with no annotation at all, reaches IDF1 88.6 and MOTA 82.4; MOTR gets 89.3 and 83.3.
What FG-ARI can't see
| Method | FG-ARI | IDF1 | Track mAP |
|---|---|---|---|
| SAVi | 90.2 | 72.3 | 42.8 |
| OC-MOT | 93.8 | 88.6 | 66.2 |
Compiled from Table 2 of the paper, CATER.
SAVi's FG-ARI is as high as 90.2, which makes its grouping look very good; yet its Track mAP, which matches over the whole video, is only 42.8. Grouping well in every frame and tracking well across the whole video are two different things.
Memory and the two attentions
| Index & merge | Memory | IDF1 | ID sw. |
|---|---|---|---|
| Dot-product similarity | rollout | 72.0 | 22,050 |
| One shared attention | rollout | 86.2 | 7,655 |
| Two attentions | Last frame only | 77.2 | 16,582 |
| Two attentions | rollout | 88.6 | 5,658 |
Compiled from Table 3 of the paper, CATER.
Remove rollout and use only the previous frame's result as the query, and ID switches climb from 5,658 to 16,582, nearly three times as many. Under occlusion, the previous frame only saw half an object, whereas rollout can piece together the whole of it.
Memory length matters too. With consecutive sampling, longer memory is better, from IDF1 82.9 at 6 frames to 88.4 at 32 frames. But the longer the memory, the more hardware it eats, so the authors switch to interleaved fast and slow sampling, which lets a 6-frame memory see long-range and occlusion cases during training as well; IDF1 reaches 88.6, higher even than 32-frame consecutive sampling.
Limitations
Both datasets are synthetic. The authors applied the same grouping module to real KITTI street scenes: the cars are detected, but the masks are coarse, especially for distant cars. DINOSAUR's feature resolution is only 1/16 of the original image, which is an architectural limit.
The paper also doesn't evaluate on MOT17. The authors explain in the appendix that MOT17 scores mainly reflect detection, and current object-centric models still can't detect as well as supervised detectors. They tried swapping the detector for SEEM and training the association on TAO in the same self-supervised way, but showed only visualizations, no numbers.
Also, the system as a whole isn't trained end to end. The object-centric model is trained first and frozen, and only then is the memory module trained.
Conclusion
OC-MOT takes on two old problems of object-centric learning in tracking. Slots that split are pieced back into one object through indexing and merging; slots that swap are held in place by memory and rollout, which carry each object's history forward.
Its most notable point is that learning the association needs no ID labels at all. Compare how alike slots and memory are, write the assignment as an expected cost, and the gradient pushes each slot back to its own seat.
The real bottleneck is the object-centric model at the front end. On CATER, OC-MOT is already close to the fully annotated MOTR, while on FISHBOWL it still trails by a few points; on real street scenes, the masks still aren't precise enough. The association part isn't picky about its detector, so a stronger front-end model plugs straight in, and the authors' attempt with SEEM in the appendix heads in that direction.