BoT-SORT: Robust Associations Multi-Pedestrian Tracking
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2206.14651
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingbotsort2022,
title = {{BoT-SORT: Robust Associations Multi-Pedestrian Tracking}},
author = {Nir Aharon and Roy Orfaig and Ben-Zion Bobrovsky},
year = {2022},
eprint = {2206.14651},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2206.14651}
}
ByteTrack showed that when the detector is strong enough, a Kalman filter plus IoU can reach a MOTA of 80.3 on MOT17.
But "IoU is enough" assumes that the box predicted from the previous frame lines up with the detection box in this frame. The moment the person filming turns around or the car turns a corner, everything in the frame moves together, and every box the Kalman filter predicts lands in the wrong place. The pedestrians haven't moved at all, yet the tracker loses a whole crowd of them.
Nir Aharon, Roy Orfaig and Ben-Zion Bobrovsky at Tel Aviv University in Israel catalog several long-standing problems with SORT-style trackers and fit the corresponding improvements onto ByteTrack one by one. The BoT in the name stands for bag of tricks, a whole bag of small tricks.
Defining the problem
The authors point out three limitations of SORT-style trackers.
The Kalman filter's state vector. SORT uses : center, area and aspect ratio. Most trackers since DeepSORT, including ByteTrack, use instead: center, aspect ratio and height. Both representations estimate the aspect ratio, and the width has to be worked back out from the aspect ratio together with the area or the height. The authors found that the width obtained this way is often inaccurate, and the box doesn't fit the person.
Camera motion. IoU matching depends entirely on where the predicted box is. When the camera moves, the overlap between predicted and detected boxes shrinks, causing ID switches or missed detections. Even with a fixed camera, wind or vibration can make the frame shake, which causes the same problems in crowded scenes.
The trade-off between IoU and appearance. Matching with IoU usually gives better MOTA; matching with re-ID features usually gives better IDF1. How do you combine the two cues without sacrificing one for the other?
Solving the problem

The overall skeleton is simply ByteTrack: high-score boxes go through a first association with all tracks, and low-score boxes go through a second association with the remaining tracks using IoU. BoT-SORT changes three things on this skeleton, and the version with re-ID added is called BoT-SORT-ReID.
1. Estimating width and height directly
The authors change the Kalman filter's state vector to include width and height directly:
The noise matrices follow DeepSORT's idea of scaling noise with box size, but now separately with width and height:
, and , which are exactly the 1/20 and 1/160 in DeepSORT's code.

In this figure, the blue dashed boxes cut off the pedestrians' feet, while the green boxes take in the whole person. When someone walks, their stride opens and closes and the width changes a lot, and a width worked back from the aspect ratio can't keep up.
The authors are very frank in the paper: this change does raise HOTA, but strictly speaking they don't know why, and can only guess that the box width fits the object more closely.
2. Camera motion compensation (CMC)
Without an IMU or navigation data, and without knowing the camera's intrinsics, image alignment between two adjacent frames is a good approximation of the camera's rigid motion projected onto the image plane.
The authors use the global motion compensation (GMC) from OpenCV's video stabilization module:
- Extract image keypoints in the previous frame.
- Track these keypoints into the current frame with sparse optical flow, and reject local outliers.
- Estimate a affine matrix with RANSAC:
is scaling and rotation, and is translation. The authors also mention that the regions covered by detection boxes contain moving objects and can be excluded from the estimate, so that it reflects only the background's motion.
With in hand, the Kalman filter's prediction is moved into the new coordinate system. The authors use a small trick: they arrange into an block-diagonal matrix , and pad with zeros into an 8-dimensional (only the two center-position dimensions have values):
Let's unpack what this does to the state vector. The eight-dimensional state can be grouped into pairs: center , width and height , center velocity , and width-and-height velocity . multiplies every pair by the same , so when the camera rotates, the center, the box size and the direction of velocity all rotate with it. is added only to the center, because camera translation only shifts an object's position; its size and velocity stay as they are. The covariance is transformed too, so the filter's uncertainty follows the new coordinate system.
Only the compensated prediction is used for IoU matching against this frame's detections.

This is from MOT17-13, where the camera is mounted on a car that's turning right. Before compensation, the whole row of predicted boxes sits off to the side of the pedestrians, and IoU matching is bound to fail; after compensation, the boxes land back on the people.
3. Fusing IoU and re-ID
Appearance features come from SBS (stronger baseline on top of BoT) in the FastReID library, with a ResNeSt50 backbone. As in StrongSORT, each track's appearance is updated with EMA, with . To keep occluded or blurry boxes from polluting the appearance, features are extracted only from high-score detections.
The common approach is a weighted sum of appearance distance and motion distance; FairMOT and StrongSORT both use . The authors use a different combination instead: filter first, then take the minimum.
is the IoU distance (1 minus IoU), and is the cosine distance between a track's appearance and a detection's feature. and .
The formula is a bit convoluted to read, so let's plug in three pairs of tracks and detection boxes:
- Close in position, similar in appearance. The IoU is 0.7, so ; the cosine distance is 0.1. Both conditions hold, , and the cost takes the smaller value, 0.05. Appearance and position agree, the cost is pushed very low, and this pair is all but certain to be matched.
- Close in position, different in appearance. IoU 0.8, ; the cosine distance is 0.4, above 0.25. Appearance doesn't count, , and the cost is 0.2. Appearance doesn't penalize the pair; it just doesn't help, and matching depends entirely on IoU.
- Similar in appearance, far in position. IoU 0.3, , above 0.5; the cosine distance is 0.05. The position is too far, so appearance doesn't count either, and the cost is 0.7.
Appearance can only help lower the cost when "the position also makes sense." Two people who look alike but are far apart won't be linked just because they're wearing the same clothes.
The authors chose the threshold after plotting the distributions of appearance distances for the same person and for different people on the MOT17 validation set: distances for the same person cluster in a very small range, while those for different people mostly lie farther out.
A debugging tool on the side: cMOTA
The authors also propose an analysis tool called current-MOTA. is simply the MOTA computed from the first frame up to frame ; computed up to the last frame, it's the ordinary MOTA.
Plot it as a curve over time, and you can see at a glance where the tracker starts going wrong.

The red line drops sharply between frames 400 and 470, which, compared with the frames above, is exactly the stretch where the camera turns. With camera motion compensation, the green line barely drops at all.
Discussion
The detector is ByteTrack's public YOLOX weights, used as is, and the re-ID model is FastReID's SBS-50, trained for 60 epochs. Tracking parameters follow ByteTrack: a score threshold of 0.6, matches with similarity below 0.2 rejected, and lost tracks kept for 30 frames. For a fair comparison, the same linear interpolation as ByteTrack's, with a maximum gap of 20 frames, is also used. Experiments run on a single RTX 3060.
What each change contributes

The Pred in the table is another small change. In high-frame-rate videos, when a track briefly goes unmatched, the Kalman filter's predicted box keeps being output instead of letting the track disappear.
Adding things up from the baseline:
- The new Kalman filter: HOTA +0.24, with MOTA and IDF1 almost unchanged.
- Camera motion compensation: MOTA +0.64, IDF1 +1.62, HOTA +0.94, the biggest step of the four.
- Prediction output: only a tiny gain on all three.
- re-ID: IDF1 another +0.54, with small rises in MOTA and HOTA.
From ByteTrack to BoT-SORT-ReID, HOTA goes up about 1.3 and IDF1 about 2.3, and more than half of that comes from camera motion compensation.
The authors also compare how the first association should use appearance features. IoU alone gives MOTA 78.4, IDF1 81.5 and HOTA 69.1; cosine distance alone drops to 73.7, 70.0, 62.4; the JDE-style "appearance distance with a Kalman filter gate" gives 77.7, 80.1, 68.2; filtering with IoU and then looking only at cosine distance gives 78.3, 81.0, 68.7. Apart from the authors' own fusion (78.5, 82.1, 69.2), every approach that uses re-ID loses to plain IoU. So the authors suggest that for applications with limited resources, IoU alone is a good choice.
Leaderboards
On the MOT17 test set:
- BoT-SORT: MOTA 80.6, IDF1 79.5, HOTA 64.6.
- BoT-SORT-ReID: MOTA 80.5, IDF1 80.2, HOTA 65.0.
The authors say BoT-SORT-ReID is the first tracker to get an IDF1 above 80 on MOT17. Compared with ByteTrack's 80.3, 77.3 and 63.1, MOTA is about the same while IDF1 and HOTA both improve clearly; ID switches fall from ByteTrack's 2196 to 1212.
On the MOT20 test set, BoT-SORT-ReID gets MOTA 77.8, IDF1 77.5 and HOTA 63.3; ByteTrack gets 77.8, 75.2 and 61.3. MOTA holds steady, and IDF1 and HOTA improve here as well.
Speed is the price. In the table, BoT-SORT runs at 6.6 FPS on MOT17 and BoT-SORT-ReID at 4.5 FPS, while ByteTrack runs at 29.6 FPS. The authors do point out, though, that each method's speed was measured on different hardware, and most don't count detection time, so FPS is hard to compare directly.
Limitations the authors admit
- Camera motion compensation can fail. When there are too many moving objects in the frame, there aren't enough background keypoints, the camera motion estimate goes wrong, and a wrong compensation can make the tracker behave unexpectedly.
- Computation time. Global motion estimation on large images takes a lot of time. The authors argue that compared with the detector's inference time this part is negligible, and it can also be parallelized with multithreading.
- Appearance features slow the tracker down. They extract features only from high-score detections to cut the cost; if needed, feature extraction can also be merged into the detector's output head, turning it into a joint architecture like JDE's.
Afterward
BoT-SORT later became the default tracker in the Ultralytics YOLO tracking module, with ByteTrack as the other option. Ultralytics' documentation recommends BoT-SORT for footage from moving cameras, such as handheld, aerial or vehicle-mounted video, and ByteTrack, with the lowest overhead, when you only need speed and the camera is fixed. Its config file makes camera motion estimation a swappable option: the default is sparse optical flow, and it can be switched to ORB, SIFT or ECC, or turned off; re-ID is optional.
Roboflow's tracker documentation gives similar advice: turn on camera motion compensation for moving cameras, and turn it off for fixed cameras if you need maximum speed; on backgrounds that lack texture, motion estimation based on sparse features may be unreliable. That's the same thing as the first limitation the paper itself lists.
Conclusion
The Kalman filter estimates width and height directly, predicted boxes move along with the camera, and IoU and appearance are filtered first and then combined by taking the minimum. That's all BoT-SORT does on top of ByteTrack. The ablations show that the biggest step is camera motion compensation; with re-ID added on top, BoT-SORT-ReID pushed MOT17 IDF1 past 80.
For video shot from a phone, a drone or a car, camera motion compensation is probably the first switch to flip on. If it's a security camera bolted to the wall, switch it off and give those few milliseconds to the detector!