MambaMOT: State-Space Model as Motion Predictor for Multi-Object Tracking
- Publication
- ICASSP 2025Conference
- First submitted to arXiv
- arXiv:2403.10826
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingmambamot2024,
title = {{MambaMOT: State-Space Model as Motion Predictor for Multi-Object Tracking}},
author = {Hsiang-Wei Huang and Cheng-Yen Yang and Wenhao Chai and Zhongyu Jiang and Jenq-Neng Hwang},
year = {2024},
eprint = {2403.10826},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2403.10826}
}
At the end of 2023, Mamba arrived as a "selective state-space model," holding its own against the Transformer on long sequences while its computation grows only linearly with sequence length. Over the next few months, people in every field tried bringing it into their own.
In multi-object tracking, the best place for a sequence model is motion prediction. Given a track's boxes over the past few frames, predict the box in the next frame: that's a sequence problem by nature, and ever since SORT it has been the Kalman filter's job.
What Hsiang-Wei Huang, Jenq-Neng Hwang and colleagues at the University of Washington did was swap the Kalman filter for Mamba. The paper is only four pages long and the idea is very direct, but it happens to make a good comparison with DiffMOT, which appeared less than two weeks earlier: the same problem, one solved with a diffusion model, the other with a state-space model.
Defining the problem
The Kalman filter assumes that over a short time an object moves in a straight line at constant speed. Pedestrians roughly fit this assumption; dancers and players don't.
OC-SORT once analyzed two weaknesses of the Kalman filter in scenes like these. One is sensitivity to state noise: if the velocity estimate is even slightly off and the object goes undetected for a few frames, the predicted position drifts far away. The other is that errors grow over time: with no new observations, the Kalman filter can only keep pushing forward along the old velocity.

The curves on the right say a lot. Most of the time the Kalman filter predicts well enough, but every time the dancer suddenly changes moves, the green line dips, falling below 0.75 several times. For IoU-based association, sudden misses like these are exactly when tracks break.
The authors' idea is that instead of assuming motion is linear, it's better to let a model learn from data how dancers usually move, and how players usually move.
Solving the problem
Predicting the next box with Mamba

MambaMOT's motion model is tiny. The input is a track's boxes over the past frames, four numbers per box; a linear projection first lifts them to 64 dimensions, they pass through two Mamba blocks, and a final MLP outputs the next frame's box. The training loss is GIoU plus mean squared error between the predicted and ground-truth boxes.
Association is taken entirely from ByteTrack: high-score boxes are first matched to the predicted boxes by IoU, and the remaining tracks are matched once more against the low-score boxes. So the only difference between MambaMOT and ByteTrack is this one component, the motion model.
What "selective" means
At the core of Mamba is a recurrence:
The hidden state is a summary of all past inputs. Each time a new input arrives, it decides how much of the old summary to keep and how much of the new input to take in.
In a traditional state-space model, and are fixed, so every step keeps and takes in the same amounts. Mamba lets them change with the input, which is why it's called "selective." The paper writes these two quantities as and , which add up to exactly 1.
Reduce this to one dimension and its meaning for motion prediction becomes clear. Let , and the recurrence becomes:
This is an exponential moving average, except that the smoothing coefficient is decided by the input.
A dancer turns around
Let's simplify the input to a dancer's horizontal velocity in each frame. For the last few frames the dancer has been moving right, 10 pixels per frame, so the memory holds . In this frame the dancer suddenly turns around, and the velocity becomes .
| Approach | Coeff. | Updated memory | Direction |
|---|---|---|---|
| Fixed | 0.2 | Still right | |
| Selective | 0.8 | Now left |
Worked example for this note; in the real model the hidden state is a high-dimensional vector, and the output also goes through an MLP.
A fixed-coefficient model treats every input the same, so it takes several frames for new information to override the old trend. A selective model can learn that when it sees this kind of input, the old trend can't be trusted and the new observation should be taken in heavily.
The Kalman filter also adjusts based on observations, but how much it adjusts is set by noise parameters chosen in advance, and underneath it's always a constant-velocity model. Mamba's way of adjusting is learned from data.
MambaMOT+: learning a track feature along the way
Since Mamba's hidden state remembers the motion pattern of the whole track, the authors want to make further use of it. MambaMOT+ attaches an extra embedding head to the output of the Mamba blocks, which outputs a feature vector for the track.
This feature is trained with a cosine embedding loss: two segments cut from the same track should have similar features, and two segments from different tracks should have their similarity pushed below 0.
After tracking finishes, the features of all tracklets are compared pairwise, and hierarchical clustering links up those with similar motion patterns. To avoid linking unreasonable combinations, pairs more than 50 frames apart in time or more than 50 pixels apart in space are not linked.
The authors stress that traditional tracklet merging needs a separately trained Siamese network that has to run once for every pair of tracklets, an computation, whereas MambaMOT+ needs only one forward pass per tracklet, . During clustering, though, the cosine similarities are still compared pairwise, so that step is still ; it's just that vector dot products are much cheaper than running a network.
Note also that this merging happens only after the whole video has been tracked, so MambaMOT+ is no longer an online method.
Training data
The motion model's training data comes from the training-set tracks of MOT17, DanceTrack and SportsMOT. Each time, a segment of length between 2 and is cut at random, padded if shorter than , and the target is the next frame's ground-truth box. It trains for 500 epochs on a single RTX 4080.
Discussion
DanceTrack and SportsMOT
| Method | HOTA | AssA | IDF1 |
|---|---|---|---|
| ByteTrack | 47.3 | 31.4 | 52.5 |
| OC-SORT | 54.6 | 40.2 | 54.6 |
| MambaMOT | 55.5 | 38.3 | 53.9 |
| MambaMOT+ | 56.1 | 39.0 | 54.9 |
Compiled from Table I of the paper, DanceTrack test set.
Compared with ByteTrack using the same set of detections, MambaMOT's HOTA is 8.2 higher, and that's the paper's main selling point.
Break it down, though, and a good chunk of that 8.2 comes from the detection side: ByteTrack's DetA is 71.6 and MambaMOT's is 80.8, while AssA is 31.4 versus 38.3. The motion model mainly affects association; with the same detections, a DetA gap of more than 9 points suggests the two setups in the table may differ in more than just the motion model. OC-SORT's DetA is 80.4, close to MambaMOT's, which makes it the fairer comparison.
Against OC-SORT, the story is rather different. OC-SORT is a method that tinkers with the Kalman filter, and MambaMOT's HOTA is only 0.9 higher, while its AssA and IDF1 are actually lower. Only MambaMOT+, with track merging added, edges ahead on IDF1, by 0.3.
SportsMOT looks similar. With the same YOLOX detector, MambaMOT's HOTA is 70.4 and OC-SORT's is 70.2; MambaMOT+ reaches 71.3. Against ByteTrack's 62.0, MambaMOT's gap is 8.4.
What's missing
The paper is very short, and leaves several things unaddressed:
- No ablations. How many input frames, how many Mamba blocks, what happens without the GIoU loss: none of it is tested. Whether an LSTM or a Transformer would do just as well isn't compared either.
- No results on pedestrian datasets. The training data includes MOT17 tracks, yet the tests report only DanceTrack and SportsMOT.
- No cross-dataset experiments. The motion model is trained on the training tracks of all three datasets together, then evaluated on the test sets of two of them. How it performs on kinds of motion it hasn't seen isn't tested.
- Incomplete comparisons. The strongest opponent in the table is OC-SORT. DiffMOT from the same period has a HOTA of 62.3 on DanceTrack and Deep OC-SORT has 61.3, and neither is listed. Those methods also use appearance features, so they can't be compared directly, but from this paper alone it's hard for a reader to judge where Mamba itself stands.
DiffMOT's ablation does offer an indirect reference. Within DiffMOT's framework, among learned motion models, an LSTM gets HOTA 51.2, a Transformer 54.6 and the diffusion model 55.7, against 46.8 for the Kalman filter. Learned motion models really are better than the Kalman filter; as for which sequence model to use, the gaps are much smaller.
Speed
MambaMOT runs at 28.8 FPS on a single GPU. The authors compare it with end-to-end methods like MOTR, which usually run at under 10 FPS. The motion model is just two 64-dimensional Mamba blocks, very light in itself.
Conclusion
MambaMOT swaps the Kalman filter for a small Mamba to see whether tracking gets better. For dancers and players, the answer is yes; compared with ByteTrack, HOTA jumps by more than 8 points in one go.
The strength of a selective state-space model is that it can decide, based on the input, whether to trust the old trend or the new observation, which is exactly what irregular motion needs most. Against improved Kalman-filter methods like OC-SORT, though, its advantage shrinks to a sliver; and whether it suits this task better than other sequence models is a question this paper leaves unanswered.