Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking
- Publication
- ICLR 2025Conference
- First submitted to arXiv
- arXiv:2410.01806
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingsamba2024,
title = {{Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking}},
author = {Mattia Segu and Luigi Piccinelli and Siyuan Li and Yung-Hsu Yang and Bernt Schiele and Luc Van Gool},
year = {2024},
eprint = {2410.01806},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2410.01806}
}
Watch a group dance and you'll notice the dancers rarely do their own thing. When the formation shifts left, everyone moves left together; when one dancer crouches, the one beside them may leap at the same moment. Flocks of birds turning and players moving into position work the same way.
For tracking, this is a very useful clue. If one dancer is blocked, we don't know exactly where that dancer is right now, but watching how the other dancers move gives a pretty good guess.
Yet most trackers handle each track separately. Kalman filters come one per track; in query-based trackers like MOTR, each query also carries only its own information from the previous frame forward. Mattia Segu, Luc Van Gool and colleagues at ETH Zurich wanted to build a model that looks at both "the long-term history of each track" and "the relationships between tracks" at once. They named it Samba, short for Synchronized Mamba, and also the samba, a dance that prizes moving in unison.
Defining the problem

The authors focus on the query-based line of tracking, the TrackFormer and MOTR family. In each frame, these methods use a set of track queries to find again, in the image, the people tracked in the previous frame, and then pass the queries on to the next frame.
This line has three problems:
- Only the previous frame counts. Queries are passed from this frame to the next, and earlier history isn't put to good use. MeMOTR added a long-term memory, but it uses an exponential moving average, so the details get averaged away.
- Every track for itself. Each query is propagated independently, and the coordination among dancers or among birds goes completely unused.
- Freezing is the only option under occlusion. When an object is blocked and detection confidence is low, MeMOTR and MOTRv2 freeze the query, holding it at the state when the object was last seen until it reappears. The longer the occlusion, the farther the frozen state drifts from the real position.
Solving the problem
Overall architecture

SambaMOTR's detector is a standard Deformable-DETR, the same as MOTR's. What changes is "how this frame's queries are passed on to the next frame."
MOTR uses a small module that turns this frame's output directly into the next frame's query. SambaMOTR swaps in Samba: each track is a sequence, and Samba uses a state-space model to compress all of the track's outputs, from its birth up to now, into a hidden state, then predicts the next frame's query from that hidden state.
Before going into Samba, the decoder output also gets a positional encoding of the box coordinates added, so the model sees both appearance and position. What Samba outputs isn't a complete new query but a residual, added to the original query.
A synchronized state-space model

The heart of Samba is "synchronization." In ordinary Mamba, each sequence's hidden state is updated only from its own input. Samba adds a synchronization at every step:
The first line is Mamba's original update, in which each track updates its memory with its own observation for this frame. The second line is the synchronization: the updated memories of all tracks are put together and run through a few layers of self-attention, letting them exchange information. The third line produces the output from the synchronized memory.
Self-attention has exactly the two properties needed: it can handle any number of tracks, and it doesn't depend on the order the tracks are listed in. The price is that computation grows with the square of the number of tracks; it's still linear in time, though, so however long a track gets, it can be followed all the way.
MaskObs: don't take in observations you can't trust
When an object is blocked, the decoder still outputs a query, but that query is mostly unreliable and may have latched onto whoever is blocking it. Use it to update the memory anyway, and the error accumulates all the way down the line.
MaskObs keeps observations with detection confidence below 0.5 out of the memory update.
Unlike freezing, the memory and the query still move forward every frame. It's just that the update now relies entirely on the track's own history, plus whatever information the synchronization brings in from other tracks that are still visible.
Let's reduce this formula to one dimension and fix and at 0.6 and 0.4 to see where the difference lies. Dancer A's memory is 1.0. In this frame A is blocked by B, and the decoder output actually looks more like B, with a value of and a confidence of only 0.3. In the next frame A reappears, with an observation of 1.0 and a confidence of 0.9.
| Approach | Occluded frame | After A reappears |
|---|---|---|
| Take it in | ||
| MaskObs |
Worked example for this note; in the real model, and change with the input, and the memory is a high-dimensional vector.
Take the observation in, and B's information gets mixed into A's memory, so even after A comes back it takes several frames to recover. With the mask, A's memory keeps most of what it had.
Training on longer clips
The MOTR family trains on 5 frames at a time, computing gradients on all 5. With only 5 frames, long-term memory can't be learned; simply lengthening the clips makes GPU memory usage grow along with them.
The authors take 10-frame clips, run the first 5 frames forward only without computing gradients, and backpropagate only through the last 5. GPU memory usage stays about the same, yet the model learns how to use a longer history.
Discussion
What each design contributes
| Setting | HOTA | AssA | IDF1 |
|---|---|---|---|
| MOTR | 54.2 | 40.2 | 51.5 |
| MeMOTR | 63.4 | 52.3 | 65.5 |
| Swap in state-space model | 63.5 | 53.8 | 67.0 |
| + MaskObs | 64.8 | 54.3 | 68.1 |
| + synchronization | 65.9 | 55.6 | 68.7 |
| + long-clip training | 67.2 | 57.5 | 70.5 |
Compiled from Table 4 of the paper, DanceTrack test set.
Just replacing the propagation module with plain Mamba already ties MeMOTR, with 1.5 more AssA. Each design added after that brings more than 1 point of HOTA, for a total of 3.8 above MeMOTR.
The appendix has a few finer-grained ablations:
- Where to synchronize. Synchronizing before adding this frame's observation gives HOTA 66.1, about the same as the 66.0 without synchronization; synchronizing after the update gives 67.2. The authors' explanation is that only when synchronization comes after the update can an occluded track use the other tracks' information from this frame to fill in the piece it's missing.
- The six hardest videos. Taking the six DanceTrack videos that do worst without synchronization, adding synchronization lifts average HOTA from 40.9 to 44.5 and AssA from 25.1 to 28.3. The messier the scene, the more synchronization helps.
- Freezing or MaskObs. Switching to MeMOTR's approach of freezing under occlusion drops HOTA from 67.2 to 65.9.
- Predicting a residual or a full query. Predicting the next frame's full query directly gives a HOTA of only 64.2; predicting the residual gives 67.2.
Three datasets
| Dataset | Strongest rival | Rival HOTA | SambaMOTR HOTA | AssA gap |
|---|---|---|---|---|
| DanceTrack | MeMOTR | 63.4 | 67.2 | +5.2 |
| BFT | OC-SORT | 66.8 | 69.6 | +4.9 |
| SportsMOT | TransTrack | 68.9 | 69.8 | +1.9 |
DanceTrack compiled from Table 1 of the paper, BFT from Table 2 of the paper, SportsMOT from Table 3 of the paper; none use extra training data.
It comes first on all three datasets, and its detector is only a ResNet-50 Deformable-DETR, not the YOLOX-X that detect-then-associate methods commonly use. On DanceTrack, C-BIoU with YOLOX-X has a DetA of 81.3, higher than SambaMOTR's 78.8, but its AssA is only 45.4, more than 12 points below SambaMOTR.
The bird-flock BFT shows the value of synchronization especially well. The birds look nearly identical, so appearance can't tell them apart, but they turn as a flock. SambaMOTR's AssA is 73.6, 12.5 higher than TrackFormer, the best query-based tracker before it.
MOT17 and other limitations
The appendix also reports MOT17. SambaMOTR's HOTA is 58.8, the same as MeMOTR's and far below ByteTrack's 63.1. The authors' view is that MOT17's pedestrians mostly walk in straight lines and there are only 7 training videos, so end-to-end methods have no edge here.
The authors list two limitations themselves. First, a track that stays inactive beyond a certain number of frames is dropped; football players often run out of view for several seconds and come back with a new ID, which calls for a separate long-term re-identification step. Second, synchronization uses self-attention, whose computation grows with the square of the number of tracks, so it slows down as crowds grow. Inference runs at 16 FPS on a single RTX 4090.
Conclusion
SambaMOTR gives query-based trackers two new things. One is long-term memory: each track uses a state-space model to compress its whole history into a hidden state, with computation linear in time. The other is synchronization: at every step the memories of all tracks exchange information, so when one track is blocked, it can take its cue from how the others are moving.
MaskObs, meanwhile, changes the handling of occlusion from "freeze" to "reject suspicious observations but keep moving forward." Together with long-clip training, the three beat earlier methods on group dances, bird flocks and ball games.
Its author list overlaps heavily with MASA's. MASA learns appearance association without any video at all; SambaMOTR learns temporal and group relationships within video. Between them, the two papers come at association from opposite ends.