Back to paper library
arXiv 2024CVPR 2024Object trackingPaper notes

DiffMOT

All in One Step

Original paper

DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction

Paper authors

Weiyi Lv · Yuhang Huang · Ning Zhang · Ruei-Sung Lin · Mei Han · Dan Zeng

Publication
CVPR 2024Conference
First submitted to arXiv
arXiv:2403.02075
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingdiffmot2024,
  title = {{DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction}},
  author = {Weiyi Lv and Yuhang Huang and Ning Zhang and Ruei-Sung Lin and Mei Han and Dan Zeng},
  year = {2024},
  eprint = {2403.02075},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.02075}
}
On this page

In the years since SORT, most trackers have used the Kalman filter as their motion model. It assumes that over a short time an object moves in a straight line at constant speed. It's fast to compute, has few parameters, and works very well on pedestrian-heavy data like MOT17.

But on the DanceTrack stage and the SportsMOT court, people stop walking nicely. Dancers suddenly turn, crouch, leap; players stop dead, change direction, sprint. Once the constant-velocity, straight-line assumption fails, the predicted box no longer lines up with the detection box, the track breaks, and a new ID pops up.

Weiyi Lv, Yuhang Huang, Dan Zeng and colleagues at Shanghai University wanted to use a diffusion model to predict these irregular motions. Diffusion models are good at fitting complex distributions; the problem is that they usually need a thousand or so steps to generate a result, while tracking has to run in real time.

Defining the problem

Original Figure 1. (a) Center trajectories of several dancers on DanceTrack over 200 frames, plotted in 3D coordinates; the trajectories twist and turn. On the right are frames 30 and 31: the top row uses the Kalman filter, and the IoU between predicted and detection boxes drops from 0.71 to 0.41, so tracking fails; the bottom row uses D²MP, with IoUs of 0.83 and 0.88. (b) HOTA and FPS of each tracker on the DanceTrack test set; DiffMOT-X has the highest HOTA, at about 23 FPS. (c) The average IoU between the Kalman filter's predicted boxes and the ground-truth boxes on four datasets: above 0.9 for MOT17 and MOT20, about 0.82 for DanceTrack, and about 0.78 for SportsMOT.
Original Figure 1, Lv et al., from PDF page 1, arXiv license, scholarly quotation.

Panel (c) on the right is where this paper starts. The authors use a Kalman filter to predict the next frame's box and compute its IoU with the ground-truth box. The average IoU is above 0.9 for both MOT17 and MOT20; DanceTrack is only about 0.82, SportsMOT about 0.78.

For tracking, when predicted boxes don't line up, association has to lean on other cues to hold together. Switch to a neural network for prediction, and simple networks like MLPs and LSTMs can't fit such complex motion; methods that use optical flow or Transformers fit it better but are too slow for real time.

DiffMOT aims to predict nonlinear motion accurately and still run fast enough.

Solving the problem

Overall architecture

DiffMOT is a standard detect-then-associate tracker, in three steps:

  1. Detection. YOLOX finds all the boxes in this frame.
  2. Motion prediction. For each existing track, use its information from the past 5 frames to predict its position in this frame. This step is the core of DiffMOT, called D²MP (decoupled diffusion-based motion predictor).
  3. Association. It reuses ByteTrack's two-stage matching: high-score detection boxes are first matched to the predicted boxes using appearance and IoU, and what's left is matched once more against the low-score boxes using IoU. For appearance, it borrows Deep OC-SORT's dynamic appearance and adaptive weighting.

Apart from step two, everything is an off-the-shelf component. So the experiments in this paper can be read as swapping the Kalman filter for a diffusion model inside the same framework.

What gets generated is the "displacement"

D²MP doesn't generate boxes; it generates how a box changes. The motion at frame ff is defined as:

Mf=Bf−Bf−1=(Δxf,Δyf,Δwf,Δhf)\begin{aligned} M_f &= B_f - B_{f-1} \\ &= (\Delta x_f, \Delta y_f, \Delta w_f, \Delta h_f) \end{aligned}

Once the motion is predicted, adding it to the previous frame's box gives this frame's predicted box.

The model learns the motion distribution of all the objects in the whole dataset, and at prediction time conditions on a single object's own history. The authors consider this more stable than fitting a separate motion model for each track: dancers' moves vary endlessly, but "how dancers move" does follow patterns.

A decoupled diffusion process

In an ordinary diffusion model, the forward process turns data into noise step by step. D²MP adopts a "decoupled diffusion" approach that splits this process into two halves running at the same time:

  1. Data decay. The clean motion data decays linearly with time tt and becomes zero at t=1t=1.
  2. Noise growth. Gaussian noise is added starting from zero and becomes pure noise at t=1t=1.

Adding the two together gives the noised motion:

Mf,t=Mf,0+t c+t z,c=−Mf,0M_{f,t} = M_{f,0} + t\,c + \sqrt{t}\,z,\qquad c = -M_{f,0}

cc is the rate of decay, and zz is standard normal noise. At t=0t=0 it's the clean motion; at t=1t=1 only zz is left.

The reverse process has to recover the motion from noise. The original decoupled diffusion needs a two-branch network that predicts cc and zz at the same time. The authors noticed that the equation above lets zz be expressed in terms of cc and Mf,tM_{f,t}, so the network only has to predict one quantity, cc.

More importantly, this process can be completed in one step.

How the one step works

Let's walk through the forward and reverse processes with an example.

Say a dancer's box in the previous frame is (300,200,80,220)(300, 200, 80, 220): center xx, center yy, width and height, in pixels. In this frame the dancer moves 12 to the right and 3 up, with the same size, so the true motion is Mf,0=(12,−3,0,0)M_{f,0} = (12, -3, 0, 0).

In the forward process, c=−Mf,0c = -M_{f,0}. At t=0.25t = 0.25:

Mf,0.25=0.75 Mf,0+0.5 z=(9,−2.25,0,0)+0.5 z\begin{aligned} M_{f,0.25} &= 0.75\,M_{f,0} + 0.5\,z \\ &= (9, -2.25, 0, 0) + 0.5\,z \end{aligned}

The motion has decayed by a quarter and is mixed with noise at half a standard deviation. By t=1t=1, only zz is left.

In reverse, it jumps from t=1t=1 straight to t=0t=0 in one step. Plugging into the reverse-process formula, the mean comes out exactly equal to −cΘ-c_\Theta, with a variance of 0. If the network infers cΘ=(−12,3,0,0)c_\Theta = (-12, 3, 0, 0) from the history, the predicted motion is (12,−3,0,0)(12, -3, 0, 0) and the predicted box is (312,197,80,220)(312, 197, 80, 220).

Worked example for this note.

An ordinary diffusion model runs hundreds to thousands of steps; in decoupled diffusion the decay is analytic and can be done in one step, so the time spent on motion prediction becomes negligible.

Where the condition comes from

To predict cΘc_\Theta, the model relies on a network called HMINet. It reads this track's information from the past 5 frames, eight numbers per frame: the four values of the box, plus that frame's displacement.

These 5 frames, together with a learnable class token, first pass through 6 layers of multi-head self-attention, and the class token that comes out the other end is the condition embedding. Next, the noised motion and the condition embedding go together into a "motion fusion layer": the condition embedding passes through an MLP to produce a scale and a shift that adjust the features of the noised motion. After a few more layers of self-attention and fusion, a final MLP outputs cΘc_\Theta.

Original Figure 3. The D²MP architecture. On the left, the motion M_f,0 is obtained by subtracting the ground-truth boxes of two adjacent frames. In the forward process at the top, the "data decay" and "noise growth" parts are added together to get the noised motion. The reverse process at the bottom is HMINet: the condition C_f (motion information from the past few frames) goes through 6 layers of multi-head self-attention to get the condition embedding E_ce, which then passes together with the noised motion through motion fusion layers and self-attention to output the predicted motion; adding it back to the previous frame's box gives this frame's predicted box. On the right are the details of multi-head self-attention and the motion fusion layer.
Original Figure 3, Lv et al., from PDF page 4, arXiv license, scholarly quotation.

During training, the loss is the smooth L1 between the predicted cΘc_\Theta and the true cc.

Discussion

Swapping out the motion model

The ablations are all on the DanceTrack validation set, with the detector and association unchanged and only the motion model swapped:

Motion modelHOTAIDF1AssA
IoU only44.736.825.3
Kalman filter46.852.131.3
LSTM51.251.634.3
Transformer54.654.638.1
D²MP55.755.239.5

Compiled from Table 5 of the paper, DanceTrack validation set.

The Kalman filter is only 2.1 HOTA better than no prediction at all; switching to D²MP adds another 8.9 over the Kalman filter. The Transformer isn't bad either, at HOTA 54.6, but the paper doesn't list its speed within the same framework.

Is one step enough?

ArchitectureStepsHOTAFPS
Two-branch144.520.9
Two-branch1052.313.1
Two-branch2054.67.5
Single-branch155.722.7

Compiled from Table 6 of the paper, DanceTrack validation set; speed includes YOLOX-X detection.

This table shows the payoff of a single branch. The two-branch version, which predicts cc and zz together, gets only 44.5 with one sampling step and needs 20 steps to reach 54.6, by which point it's down to 7.5 FPS. The single-branch version, which predicts only cc, gets 55.7 in a single step. With one fewer quantity to learn, the network actually learns better.

How much history?

For the history length nn used as the condition, going from 1 frame to 10 frames, HOTA is 51.7, 52.5, 53.0, 55.7 (5 frames), 52.5 and 51.1 in turn. Too short, and you can't tell where the object is heading; too long, and the old information actually gets in the way of prediction. Five frames is best.

Three kinds of condition content were also tried: only the previous frame's box, only the previous frame's displacement, or both. Using both gives the highest HOTA, though the three differ by at most 1.3.

Test-set results

DatasetDiffMOTBest otherMethod
DanceTrack62.361.3Deep OC-SORT
SportsMOT76.274.1MixSort-OC
MOT1764.565.1SparseTrack
MOT2061.763.9Deep OC-SORT

Test-set HOTA. DanceTrack compiled from Table 1 of the paper; SportsMOT from Table 2 of the paper, with the detector trained on the training plus validation sets; MOT17 from Table 3 of the paper; MOT20 from Table 9 in the paper's appendix.

On DanceTrack and SportsMOT, where nonlinear motion dominates, DiffMOT is first on both. On DanceTrack, its IDF1 of 63.0 and AssA of 47.2 are also 1.5 and 1.4 higher than Deep OC-SORT, so the gain is mainly in association.

On the pedestrian-heavy MOT17 and MOT20, DiffMOT is no longer the best, a little behind Deep OC-SORT and SparseTrack. Pedestrian motion is close to a straight line to begin with, the Kalman filter already does the job, and the diffusion model has little room to show what it can do.

As for speed, it runs at 22.7 FPS with YOLOX-X; switching to YOLOX-S gets it to 30.3 FPS, while HOTA drops to 53.3.

Across datasets

In the appendix, the authors take the D²MP trained on DanceTrack and apply it directly to other datasets. On SportsMOT it gets HOTA 75.6, only 0.6 below the version trained on SportsMOT, and still above MixSort-OC's 74.1. On MOT17 it's 2.7 lower.

Judging by the results, the motion of dancers and players is closer to each other than to that of pedestrians.

When it fails

The appendix lists two kinds of failure. The first is two people swapping IDs as they cross paths, which the authors attribute to the model having no constraint on the direction of velocity. The second is that people who disappear for a long time can't be recovered: D²MP only predicts one frame ahead at a time, so for a track that has been broken for a long while, it has no way of pushing forward several frames to wait for the person.

Conclusion

DiffMOT changes only one part, replacing the Kalman filter in the detect-then-associate framework with a diffusion model.

The key to making it usable for real-time tracking is decoupled diffusion. The data decay is analytic, so the reverse process can be done in one step; the network only has to predict the decay rate cc, which is easier to learn than predicting cc and the noise together. The result is a predictor that can fit the irregular motion of dancers and players and, detection included, still runs at over 20 FPS.

Compared with DiffusionTrack from half a year earlier, both papers use diffusion models: DiffusionTrack uses one to generate paired boxes for two frames, DiffMOT only to generate displacements. On DanceTrack, the latter's HOTA is higher by nearly 10.