Back to paper library
arXiv 2024ICASSP 2025Object trackingPaper notes

MambaMOT

Mamba Mentality

Original paper

MambaMOT: State-Space Model as Motion Predictor for Multi-Object Tracking

Paper authors

Hsiang-Wei Huang · Cheng-Yen Yang · Wenhao Chai · Zhongyu Jiang · Jenq-Neng Hwang

Publication
ICASSP 2025Conference
First submitted to arXiv
arXiv:2403.10826
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingmambamot2024,
  title = {{MambaMOT: State-Space Model as Motion Predictor for Multi-Object Tracking}},
  author = {Hsiang-Wei Huang and Cheng-Yen Yang and Wenhao Chai and Zhongyu Jiang and Jenq-Neng Hwang},
  year = {2024},
  eprint = {2403.10826},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10826}
}
On this page

At the end of 2023, Mamba arrived as a "selective state-space model," holding its own against the Transformer on long sequences while its computation grows only linearly with sequence length. Over the next few months, people in every field tried bringing it into their own.

In multi-object tracking, the best place for a sequence model is motion prediction. Given a track's boxes over the past few frames, predict the box in the next frame: that's a sequence problem by nature, and ever since SORT it has been the Kalman filter's job.

What Hsiang-Wei Huang, Jenq-Neng Hwang and colleagues at the University of Washington did was swap the Kalman filter for Mamba. The paper is only four pages long and the idea is very direct, but it happens to make a good comparison with DiffMOT, which appeared less than two weeks earlier: the same problem, one solved with a diffusion model, the other with a state-space model.

Defining the problem

The Kalman filter assumes that over a short time an object moves in a straight line at constant speed. Pedestrians roughly fit this assumption; dancers and players don't.

OC-SORT once analyzed two weaknesses of the Kalman filter in scenes like these. One is sensitivity to state noise: if the velocity estimate is even slightly off and the object goes undetected for a few frames, the predicted position drifts far away. The other is that errors grow over time: with no new observations, the Kalman filter can only keep pushing forward along the old velocity.

Original Fig. 1. On the left are two frames from DanceTrack; the top row shows MambaMOT's predicted boxes, the bottom row ByteTrack's (Kalman filter) predicted boxes, and the red boxes are the ground truth; the Kalman filter's green boxes are offset from the ground truth. On the right is the per-frame IoU between the predicted box and the ground truth for the same target: the blue line is Mamba, averaging 0.95; the green line is the Kalman filter, averaging 0.86 and often dropping suddenly below 0.75.
Original Fig. 1, Huang et al., from PDF page 1, arXiv license, scholarly quotation.

The curves on the right say a lot. Most of the time the Kalman filter predicts well enough, but every time the dancer suddenly changes moves, the green line dips, falling below 0.75 several times. For IoU-based association, sudden misses like these are exactly when tracks break.

The authors' idea is that instead of assuming motion is linear, it's better to let a model learn from data how dancers usually move, and how players usually move.

Solving the problem

Predicting the next box with Mamba

Original Fig. 2. On the left is the MambaMOT+ architecture. The boxes of one track in each frame go through a linear projection and into two Mamba blocks, with the hidden state passed forward frame by frame. The output at the last frame feeds two heads: the prediction head outputs the next frame's box, and the embedding head outputs a feature vector for this track. On the right is the inside of a Mamba block: an MLP expands the dimension and splits into two paths, one through a 1D convolution and a selective SSM, the other acting as a gate; the two are multiplied and projected back to the original dimension.
Original Fig. 2, Huang et al., from PDF page 3, arXiv license, scholarly quotation.

MambaMOT's motion model is tiny. The input is a track's boxes over the past nn frames, four numbers per box; a linear projection first lifts them to 64 dimensions, they pass through two Mamba blocks, and a final MLP outputs the next frame's box. The training loss is GIoU plus mean squared error between the predicted and ground-truth boxes.

Association is taken entirely from ByteTrack: high-score boxes are first matched to the predicted boxes by IoU, and the remaining tracks are matched once more against the low-score boxes. So the only difference between MambaMOT and ByteTrack is this one component, the motion model.

What "selective" means

At the core of Mamba is a recurrence:

ht=Aˉt ht−1+Bˉt xt,yt=C hth_t = \bar{A}_t\, h_{t-1} + \bar{B}_t\, x_t,\qquad y_t = C\, h_t

The hidden state hth_t is a summary of all past inputs. Each time a new input arrives, it decides how much of the old summary to keep and how much of the new input to take in.

In a traditional state-space model, Aˉ\bar{A} and Bˉ\bar{B} are fixed, so every step keeps and takes in the same amounts. Mamba lets them change with the input, which is why it's called "selective." The paper writes these two quantities as Aˉt=1−σ(Linear(xt))\bar{A}_t = 1 - \sigma(\text{Linear}(x_t)) and Bˉt=σ(Linear(xt))\bar{B}_t = \sigma(\text{Linear}(x_t)), which add up to exactly 1.

Reduce this to one dimension and its meaning for motion prediction becomes clear. Let st=σ(Linear(xt))s_t = \sigma(\text{Linear}(x_t)), and the recurrence becomes:

ht=(1−st) ht−1+st xth_t = (1 - s_t)\, h_{t-1} + s_t\, x_t

This is an exponential moving average, except that the smoothing coefficient sts_t is decided by the input.

A dancer turns around

Let's simplify the input to a dancer's horizontal velocity in each frame. For the last few frames the dancer has been moving right, 10 pixels per frame, so the memory holds h=10h = 10. In this frame the dancer suddenly turns around, and the velocity becomes −10-10.

ApproachCoeff.Updated memoryDirection
Fixed0.20.8×10+0.2×(−10)=60.8 \times 10 + 0.2 \times (-10) = 6Still right
Selective0.80.2×10+0.8×(−10)=−60.2 \times 10 + 0.8 \times (-10) = -6Now left

Worked example for this note; in the real model the hidden state is a high-dimensional vector, and the output also goes through an MLP.

A fixed-coefficient model treats every input the same, so it takes several frames for new information to override the old trend. A selective model can learn that when it sees this kind of input, the old trend can't be trusted and the new observation should be taken in heavily.

The Kalman filter also adjusts based on observations, but how much it adjusts is set by noise parameters chosen in advance, and underneath it's always a constant-velocity model. Mamba's way of adjusting is learned from data.

MambaMOT+: learning a track feature along the way

Since Mamba's hidden state remembers the motion pattern of the whole track, the authors want to make further use of it. MambaMOT+ attaches an extra embedding head to the output of the Mamba blocks, which outputs a feature vector for the track.

This feature is trained with a cosine embedding loss: two segments cut from the same track should have similar features, and two segments from different tracks should have their similarity pushed below 0.

After tracking finishes, the features of all tracklets are compared pairwise, and hierarchical clustering links up those with similar motion patterns. To avoid linking unreasonable combinations, pairs more than 50 frames apart in time or more than 50 pixels apart in space are not linked.

The authors stress that traditional tracklet merging needs a separately trained Siamese network that has to run once for every pair of tracklets, an O(N2)O(N^2) computation, whereas MambaMOT+ needs only one forward pass per tracklet, O(N)O(N). During clustering, though, the cosine similarities are still compared pairwise, so that step is still O(N2)O(N^2); it's just that vector dot products are much cheaper than running a network.

Note also that this merging happens only after the whole video has been tracked, so MambaMOT+ is no longer an online method.

Training data

The motion model's training data comes from the training-set tracks of MOT17, DanceTrack and SportsMOT. Each time, a segment of length between 2 and nn is cut at random, padded if shorter than nn, and the target is the next frame's ground-truth box. It trains for 500 epochs on a single RTX 4080.

Discussion

DanceTrack and SportsMOT

MethodHOTAAssAIDF1
ByteTrack47.331.452.5
OC-SORT54.640.254.6
MambaMOT55.538.353.9
MambaMOT+56.139.054.9

Compiled from Table I of the paper, DanceTrack test set.

Compared with ByteTrack using the same set of detections, MambaMOT's HOTA is 8.2 higher, and that's the paper's main selling point.

Break it down, though, and a good chunk of that 8.2 comes from the detection side: ByteTrack's DetA is 71.6 and MambaMOT's is 80.8, while AssA is 31.4 versus 38.3. The motion model mainly affects association; with the same detections, a DetA gap of more than 9 points suggests the two setups in the table may differ in more than just the motion model. OC-SORT's DetA is 80.4, close to MambaMOT's, which makes it the fairer comparison.

Against OC-SORT, the story is rather different. OC-SORT is a method that tinkers with the Kalman filter, and MambaMOT's HOTA is only 0.9 higher, while its AssA and IDF1 are actually lower. Only MambaMOT+, with track merging added, edges ahead on IDF1, by 0.3.

SportsMOT looks similar. With the same YOLOX detector, MambaMOT's HOTA is 70.4 and OC-SORT's is 70.2; MambaMOT+ reaches 71.3. Against ByteTrack's 62.0, MambaMOT's gap is 8.4.

What's missing

The paper is very short, and leaves several things unaddressed:

  1. No ablations. How many input frames, how many Mamba blocks, what happens without the GIoU loss: none of it is tested. Whether an LSTM or a Transformer would do just as well isn't compared either.
  2. No results on pedestrian datasets. The training data includes MOT17 tracks, yet the tests report only DanceTrack and SportsMOT.
  3. No cross-dataset experiments. The motion model is trained on the training tracks of all three datasets together, then evaluated on the test sets of two of them. How it performs on kinds of motion it hasn't seen isn't tested.
  4. Incomplete comparisons. The strongest opponent in the table is OC-SORT. DiffMOT from the same period has a HOTA of 62.3 on DanceTrack and Deep OC-SORT has 61.3, and neither is listed. Those methods also use appearance features, so they can't be compared directly, but from this paper alone it's hard for a reader to judge where Mamba itself stands.

DiffMOT's ablation does offer an indirect reference. Within DiffMOT's framework, among learned motion models, an LSTM gets HOTA 51.2, a Transformer 54.6 and the diffusion model 55.7, against 46.8 for the Kalman filter. Learned motion models really are better than the Kalman filter; as for which sequence model to use, the gaps are much smaller.

Speed

MambaMOT runs at 28.8 FPS on a single GPU. The authors compare it with end-to-end methods like MOTR, which usually run at under 10 FPS. The motion model is just two 64-dimensional Mamba blocks, very light in itself.

Conclusion

MambaMOT swaps the Kalman filter for a small Mamba to see whether tracking gets better. For dancers and players, the answer is yes; compared with ByteTrack, HOTA jumps by more than 8 points in one go.

The strength of a selective state-space model is that it can decide, based on the input, whether to trust the old trend or the new observation, which is exactly what irregular motion needs most. Against improved Kalman-filter methods like OC-SORT, though, its advantage shrinks to a sliver; and whether it suits this task better than other sequence models is a question this paper leaves unanswered.