DiffusionTrack: Diffusion Model For Multi-Object Tracking
- Publication
- AAAI 2024Conference
- First submitted to arXiv
- arXiv:2308.09905
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingdiffusiontrack2023,
title = {{DiffusionTrack: Diffusion Model For Multi-Object Tracking}},
author = {Run Luo and Zikai Song and Lintao Ma and Jinlin Wei and Wei Yang and Min Yang},
year = {2023},
eprint = {2308.09905},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2308.09905}
}
Diffusion models are best known for generating images. They start from pure noise and remove it step by step until a clear image emerges.
At the end of 2022, DiffusionDet carried this idea over to object detection. Instead of images, it generates boxes, scattering a pile of random boxes over the image and letting the network correct them step by step until they land on objects.
Run Luo, Zikai Song and colleagues at the Shenzhen Institute of Advanced Technology and Huazhong University of Science and Technology wanted to go one step further. If what's generated is not one box but a pair of boxes, one in the previous frame and one in this frame, both framing the same person, then detection and tracking get done together in a single denoising process.
Defining the problem
The authors divide the trackers of the time into two families and argue that they share three problems:
- Inconsistency. In detect-then-track methods, the detector and tracker are trained separately. Joint detection-and-tracking methods put both in one network, but detection and tracking still go down separate branches.
- A hard trade-off between robustness and complexity. Detect-then-track methods are simple in structure, but as soon as detection is off, tracking goes wrong with it. Joint methods are more stable but complex, and their detection accuracy can't match that of dedicated detectors.
- Inflexibility. Within the same video, some stretches have few people and some are crowded, some people walk slowly and some run, yet most methods can use only one set of settings for the entire video.
DiffusionTrack aims to handle all three at once with one unified framework.
Solving the problem
Generating a pair of boxes at once

The unit of data DiffusionTrack works with is a pair of boxes, the same object's boxes in frame and frame , eight numbers in all. The network's job is to recover the true pair from a pair scrambled by noise.
If the two frames are the same image, the problem degenerates into ordinary object detection. The authors stress this, because it shows that detection and tracking really are trained in the same model under the same objective.
The network has two parts:
- Backbone. ByteTrack's pretrained YOLOX plus an FPN, run once on each of the two frames and then frozen.
- Diffusion head. For each pair of noise boxes, it takes an RoI feature from each frame's feature map and refines them repeatedly through several blocks. Each block outputs the box positions, the class, and an association score indicating whether the pair of boxes is the same object.
Each block contains a spatial-temporal fusion module that lets the two paired boxes exchange information. The box in the previous frame can see this frame, and the box in this frame can see the previous one, so the two sides decide together which way to correct the pair.
Training: dirtying the right answers
During training, two frames are taken from the same video, at most 5 frames apart. Each object that appears in both frames forms a correct pair of boxes, and extra boxes are padded in to make up 500 pairs.
Then, following the diffusion schedule, varying amounts of Gaussian noise are added to these 500 pairs, and the network learns to restore them. Matching uses the Hungarian algorithm, and the loss has three terms: focal loss for the class, L1 for the coordinates, and a 3D GIoU term.
3D GIoU is an extension of GIoU. A pair of boxes stacked along the time axis looks like a frustum; the intersection and union of two frustums are computed by summing the areas over the two frames. This makes the network attend to the boxes in both frames together instead of scoring each one separately.
Inference: starting from the previous frame's boxes
Detection starts from purely random boxes; tracking doesn't have to, because the previous frame already tells you where everyone is.

Inference takes three steps:
- Pad the boxes. Repeat the previous frame's boxes several times and fill the remaining slots with random Gaussian boxes, up to a fixed number.
- Add noise. Mix in noise according to .
- Denoise. Run a few DDIM steps to get the paired boxes.
Here's an example. On MOT17 the setting is 800 boxes, 25% of them taken from the previous frame. If 40 people were tracked in the previous frame, that's 200 prior boxes, with each person repeated 5 times; the remaining 600 are random boxes.
The pairs that start from prior boxes carry people already being tracked into this frame; the pairs that start from random boxes look for people who have just appeared, or pick up people lost earlier.
Worked example for this note.
After denoising, pairs with an association score below 0.25 are discarded first, then two rounds of NMS remove duplicates. The remaining pairs are linked to existing tracks by IoU. For tracks lost to occlusion, a simple Kalman filter estimates their position, and they're matched once more against the newly found boxes.
How much noise to add depends on how fast objects move. The authors make the diffusion time step vary with the average motion between the two frames. The more movement, the more noise, giving the network more room to make corrections. Different stretches of the same video can therefore use different amounts of noise, which is the "flexibility" mentioned earlier.
Discussion
How much prior, what kind of noise
The ablations all train on the first half of the MOT17 training set and test on the second half:
| Prior share | MOTA | IDF1 | HOTA |
|---|---|---|---|
| 0% | 71.2 | 65.9 | 58.1 |
| 25% | 73.6 | 70.0 | 60.7 |
| 50% | 74.5 | 71.2 | 61.8 |
| 75% | 74.1 | 71.4 | 61.9 |
| 100% | 72.9 | 66.8 | 58.4 |
Compiled from Table 1(a) of the paper, MOT17 validation set.
0% means using no information from the previous frame at all and detecting from scratch every frame; 100% means padding with no random boxes at all. Both extremes are worse, and the proportions in between do best. The former lacks tracking cues, and the latter has no random boxes to find newly appearing people.
The kind of noise used for padding matters too. Gaussian noise is best, with a HOTA of 60.7; switching to Poisson or uniform noise, or simply padding with image-sized boxes, drops HOTA below 59 in every case.
For the relation between noise amount and motion, a logarithmic function works best, with a HOTA of 62.6; a linear function gives 60.7, and a fixed value 58.2.
More boxes, more steps
| Boxes | Steps | MOTA | HOTA | FLOPs | FPS |
|---|---|---|---|---|---|
| 500 | 1 | 71.5 | 58.4 | 229.6G | 21.05 |
| 500 | 2 | 71.7 | 59.5 | 459.2G | 10.47 |
| 800 | 1 | 73.6 | 60.7 | 367.3G | 15.89 |
| 1000 | 1 | 74.1 | 61.3 | 459.1G | 13.37 |
Compiled from Table 1(d) of the paper, MOT17 validation set, a single RTX 3090, FP16.
One advantage of diffusion models is that once training is done, both the number of boxes and the number of denoising steps can be adjusted at inference. More boxes and more steps mean slightly better results and slightly lower speed, with no retraining.
From the table, though, adding boxes is a better use of compute than adding steps. At the same compute of about 459G, 1000 boxes with 1 step reach a HOTA of 61.3, while 500 boxes with 2 steps get only 59.5. In the appendix, the authors also admit that multi-step denoising brings limited gains.
When detection is disturbed

This is the paper's most striking figure. The authors mix noise into the detection boxes, raising the ratio from 0 to 0.1. ByteTrack, which depends entirely on the detection boxes, already has a negative MOTA at a perturbation of 0.04; DiffusionTrack is barely affected.
The result isn't hard to understand. For DiffusionTrack, noisy boxes are its input to begin with; denoising is what it does all day.
Mind the experimental setup when reading this figure. According to the caption, all four trackers are trained on the MOT17 training set and then evaluated on val-half, and val-half is simply the second half of the training set. That's why DiffusionTrack's unperturbed MOTA is close to 89, far above the 73.6 in the earlier ablations. What can be compared is the shape of the curves; the values themselves can't be set against the other tables.
Compared with other methods
| Dataset | Method | MOTA | IDF1 | HOTA | AssA |
|---|---|---|---|---|---|
| MOT17 | ByteTrack | 80.3 | 77.3 | 63.1 | 62.0 |
| MOT17 | DiffusionTrack | 77.9 | 73.8 | 60.8 | 58.8 |
| MOT20 | ByteTrack | 77.8 | 75.2 | 61.3 | 59.6 |
| MOT20 | DiffusionTrack | 72.8 | 66.3 | 55.3 | 51.3 |
| DanceTrack | OC-SORT | 89.6 | 54.6 | 54.6 | 40.2 |
| DanceTrack | DiffusionTrack | 89.3 | 47.5 | 52.4 | 33.5 |
MOT17 and MOT20 compiled from Table 2 of the paper, DanceTrack from Table 3; all test sets, private detections.
Among joint detection-and-tracking methods, DiffusionTrack is the best in the paper's table, with an MOT17 MOTA of 77.9, above TrackFormer's 74.1 and MOTR's 71.9. Against the detect-then-track ByteTrack and OC-SORT, though, it still trails on all three datasets.
Broken down, DiffusionTrack's strength is detection. On DanceTrack its DetA is 82.2, even higher than OC-SORT's 80.4, but its AssA is only 33.5, 6.7 below OC-SORT. On MOT17 it has 3,819 ID switches, against only 2,196 for ByteTrack.
The authors also built a baseline version that adds noise only in frame . On MOT17 its HOTA is 55.9, 4.9 below DiffusionTrack; on crowded MOT20 it's only 42.5, and on DanceTrack, with its complex motion, only 44.0. The authors' explanation is that the baseline only learns to regress from the previous frame's box to this frame, whereas adding noise to both frames together forces the network to learn to tell different objects apart.
The cost
The diffusion head has 236 million trainable parameters and takes about 30 hours to train on 8 RTX 3090s. At inference, 800 boxes with one step cost 367G of computation per frame, about 16 FPS. The authors also mention in the appendix that small objects in MOT20 aren't handled well.
Conclusion
DiffusionTrack casts tracking as a denoising problem, with the same person's boxes in two frames gradually refined out of a pair of noise boxes. Detection and tracking are trained with the same network and the same objective; the number of boxes and denoising steps can be tuned at inference; and it holds steady when the detection boxes are disturbed.
Its weakness is association. Its detection metrics are very good, but its identity metrics still can't catch up with several methods that rely on a Kalman filter and IoU, and in the end it uses IoU and a Kalman filter itself to link the tracks.
Diffusion models have another use in tracking. Half a year later, DiffMOT used one not to generate boxes but to predict where a person will go next.