Back to paper library
arXiv 2021ECCV 2022Object trackingPaper notes

ByteTrack

Not One Left Behind

Original paper

ByteTrack: Multi-Object Tracking by Associating Every Detection Box

Paper authors

Yifu Zhang · Peize Sun · Yi Jiang …

Show all 9 authors

Yifu Zhang · Peize Sun · Yi Jiang · Dongdong Yu · Fucheng Weng · Zehuan Yuan · Ping Luo · Wenyu Liu · Xinggang Wang

Publication
ECCV 2022Conference
First submitted to arXiv
arXiv:2110.06864
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingbytetrack2021,
  title = {{ByteTrack: Multi-Object Tracking by Associating Every Detection Box}},
  author = {Yifu Zhang and Peize Sun and Yi Jiang and Dongdong Yu and Fucheng Weng and Zehuan Yuan and Ping Luo and Wenyu Liu and Xinggang Wang},
  year = {2021},
  eprint = {2110.06864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.06864}
}
On this page

The introduction of this paper opens with a line from Hegel: "What is reasonable is real; that which is real is reasonable."

Tracking papers don't often quote philosophers. What the authors want to say is that some of the low-score boxes a detector outputs are actually real.

Ever since SORT, almost every tracker has had the same habit. First throw away boxes whose detection score is below some threshold, then match only the high-score boxes. The reasoning makes sense: low-score boxes contain a lot of background and false positives, and keeping them only interferes with tracking.

But when a person is half hidden behind someone else, the detector's score drops too. Throw those boxes away and that person's track breaks.

Yifu Zhang, first author of FairMOT, this time working with a team from Huazhong University of Science and Technology, the University of Hong Kong and ByteDance, shifts attention away from network architecture and onto this threshold.

Defining the problem

Original Figure 2. A tracking example over three consecutive frames, t1, t2 and t3. (a) Detection boxes and scores: the woman on the left is always 0.9, the elderly person right of center is 0.9 and 0.8, the person hidden behind the woman drops from 0.8 to 0.4 and then to 0.1, and there is always a 0.1 box on the background region at the far right. (b) Associating only detection boxes scoring above 0.5: the occluded person (red box) disappears at t2 and t3. (c) Associating every detection box: the occluded person is reconnected to the original red track at t2 and t3 through low-score boxes, with the dashed lines showing the Kalman filter's predicted boxes; the background box at the far right is discarded because no track overlaps it.
Original Figure 2, Zhang et al., from PDF page 2, arXiv license, scholarly quotation.

Let's start with row (a). The first frame has four boxes. The three people all score above 0.8, and the 0.1 box at the far right is background.

In the second frame, the woman on the left walks in front of the person in the middle and partly blocks them, and their score drops from 0.8 to 0.4. In the third frame, the occlusion is worse and the score is down to 0.1.

(b) is the usual approach, matching only boxes that score above 0.5. The blocked person loses their observation in the second frame, and the red track disappears with it. When they walk out from behind the woman, they'll be treated as a new person.

What if we keep everything? Then the 0.1 background box at the far right is kept too, and becomes a fake track.

That's the dilemma of the detection threshold. Set it high and you miss occluded people; set it low and you bring in background false positives. The authors point out that few methods really deal with this dilemma.

The clue they noticed is similarity to existing tracks. The blocked person has a very low score, but their box overlaps heavily with where the Kalman filter predicts the red track to be; the background box at the far right has no track anywhere near it. Just compare the low-score boxes against "the tracks left unmatched in the previous round," and you can recover the real objects while filtering out the background.

(c) is the result of doing exactly that.

Solving the problem

The authors name this association method BYTE. The name comes from the byte in computing: each detection box is a basic unit of a track, just as a byte is a basic unit of a program.

The whole procedure has only two rounds of association:

Setup: Use the detection score threshold τ\tau to split this frame's detection boxes into two groups, high-score boxes Dhigh\mathcal{D}_{high} and low-score boxes Dlow\mathcal{D}_{low}. Use the Kalman filter to predict where each track will be in this frame.

First association: Match the high-score boxes against all tracks, including temporarily lost ones. Similarity can be the IoU between predicted boxes and detection boxes, or the distance between re-ID features, solved with the Hungarian algorithm. Unmatched high-score boxes are recorded as Dremain\mathcal{D}_{remain}, and unmatched tracks as Tremain\mathcal{T}_{remain}.

Second association: Match the low-score boxes against Tremain\mathcal{T}_{remain}, using IoU only as the similarity. After this step, any low-score boxes still unmatched are thrown away as background.

Wrap-up: Tracks unmatched in both rounds move to the lost set and are kept for 30 frames. If one is matched in the first association during that time, it gets its original ID back; only after 30 frames is it deleted. High-score boxes left unmatched in the first association start new tracks. Lost tracks aren't output.

Why does the second association use only IoU? Low-score boxes usually have low scores because of heavy occlusion or motion blur, and appearance features are unreliable at those moments. Position is actually more reliable: a blocked person can't get very far in one frame.

BYTE is independent of the detector and doesn't prescribe which similarity the first association uses, so it can be bolted onto the back of any tracker.

The authors then attach BYTE to YOLOX, a very strong detector at the time, and call the result ByteTrack. On pedestrian datasets like MOT17 and MOT20, ByteTrack uses only IoU in both rounds of association, with no re-ID features at all.

Discussion

ByteTrack's detector is YOLOX-X. After pretraining on COCO, it's trained for 80 epochs on MOT17, CrowdHuman, CityPersons and ETHZ together, with an input size of 1440×8001440 \times 800, taking about 12 hours on 8 V100s. Speed is measured on a V100 with FP16 and a batch size of 1.

For the ablations, each video in the MOT17 training set is cut into two halves, with the first half used for training and the second for validation, and CrowdHuman is added to the training data.

Against other association methods

Original Table 2. A comparison of SORT, DeepSORT, MOTDT and BYTE (without or with re-ID) on the MOT17 and BDD100K validation sets. On MOT17, SORT has MOTA 74.6, IDF1 76.9 and 291 ID switches; DeepSORT 75.4, 77.2, 239; MOTDT 75.8, 77.6, 273; BYTE 76.6, 79.3, 159; BYTE with re-ID 76.3, 80.5, 216. On BDD100K, mMOTA is, in the same order, 30.9, 24.5, 26.7, 39.4 and 45.5. FPS is, in the same order, 30.1, 13.5, 11.1, 29.6 and 11.8.
Original Table 2, Zhang et al., from PDF page 6, arXiv license, scholarly quotation.

Every row of this table uses the same detector; only the association method differs.

SORT can be seen as BYTE's baseline. Both use only a Kalman filter and IoU, and the only difference is whether there's a second association. BYTE raises MOTA from 74.6 to 76.6 and IDF1 from 76.9 to 79.3, and cuts ID switches from 291 to 159, at almost the same speed (30.1 vs. 29.6 FPS). That's what low-score boxes are worth.

What surprised the authors more was DeepSORT. It has an extra re-ID model, yet it loses to the IoU-only BYTE on MOTA, IDF1 and ID switches, and it runs at less than half the speed. The authors' reading is that when detection boxes are accurate enough, a simple Kalman filter can handle long-range association well; under heavy occlusion, re-ID features tend to go wrong, and the motion model is more reliable.

With re-ID added, BYTE's IDF1 rises to 80.5, but MOTA dips slightly and ID switches go up.

BDD100K is a different story. These are dashcam videos with large camera motion and a low annotation frame rate, where the Kalman filter fails, so the authors don't use a Kalman filter on BDD100K. IoU-only BYTE is far ahead of SORT (39.4 vs. 30.9), but its ID switches reach 27902; with re-ID added, mMOTA rises to 45.5 and ID switches fall to 9140.

The authors also compare which similarity each of the two associations should use. On MOT17, either IoU or re-ID works for the first association: IoU gives better MOTA and ID switches, and re-ID gives higher IDF1. On BDD100K, re-ID is much better for the first association. But on both datasets IoU is better for the second association, by about 1 point of MOTA over re-ID. This supports the judgment that appearance features of low-score boxes are unreliable.

Less sensitive to the threshold

Original Figure 3. MOTA and IDF1 of BYTE (red line) and SORT (blue line) on the MOT17 validation set as the detection score threshold goes from 0.2 to 0.8. BYTE's MOTA stays around 76 from 0.2 to 0.7 and only drops to about 74 at 0.8; SORT's MOTA peaks at about 74.5 around 0.4 and then falls steadily to only about 66 at 0.8. IDF1 shows a similar trend: BYTE is mostly between 78 and 79 and drops to about 77 at 0.8; SORT falls to about 73 at 0.8.
Original Figure 3, Zhang et al., from PDF page 6, arXiv license, scholarly quotation.

The detection threshold is usually a tracker's most sensitive hyperparameter, the one that needs the most careful tuning. The authors vary it from 0.2 to 0.8. SORT's MOTA slides quickly as the threshold rises, from about 74.5 at 0.4 to about 66 at 0.8; BYTE stays comparatively steady.

The reason is straightforward. BYTE's second association recovers boxes below the threshold, so when the threshold goes up, the boxes that get thrown away still have a second chance.

The authors also count the true positives and false positives among the low-score boxes. Among all boxes scoring between the lower bound and the threshold, some videos (MOT17-02, for example) have far more false positives than true positives; but the boxes BYTE recovers from that pool contain clearly more true positives than false positives. Similarity to existing tracks really is an effective filter.

Bolted onto other trackers

Original Table 3. Results on the MOT17 validation set when BYTE is applied to nine trackers. Most trackers have three rows each: the original method, BYTE's second association inserted into the original association procedure, and full BYTE run directly on that tracker's detection boxes (Motion(K) means Kalman filter). For example, FairMOT goes from MOTA 69.1, IDF1 72.8 and 299 ID switches to 70.4, 74.2, 232 and 70.3, 73.2, 236; CenterTrack from 66.1, 64.2, 528 to 66.3, 64.8, 334 and 67.4, 74.0, 144; MOTR from 64.7, 67.2, 346 to 64.3, 69.3, 263 and 65.7, 68.4, 260. Green marks items that improve by at least 1 point.
Original Table 3, Zhang et al., from PDF page 7, arXiv license, scholarly quotation.

This is the most convincing table in the whole paper. The authors apply BYTE to nine very different trackers: methods that combine motion and re-ID, such as JDE, CSTrack, FairMOT and TraDeS; QDTrack, which uses re-ID only; CenterTrack, with learned motion; Chained-Tracker, with its chained structure; and TransTrack and MOTR, which propagate boxes with attention.

There are two ways to apply it:

  1. Insertion mode: the original association runs as usual, and afterward the unmatched tracks are matched once more against the low-score boxes using IoU.
  2. Replacement mode: only the tracker's detection boxes are kept, and the association is replaced entirely with BYTE.

Almost every tracker improves on almost every metric. The most striking is CenterTrack: in replacement mode, MOTA goes up 1.3 and IDF1 up 9.8, and ID switches fall from 528 to 144. CenterTrack originally matches locally only by working backward from offsets, so the moment a blocked person's track breaks, the ID changes. BYTE's mechanism for keeping lost tracks fills exactly that gap.

Still, it isn't an improvement across the board. TraDeS loses 0.3 MOTA in replacement mode; MOTR loses 0.4 MOTA in insertion mode, while gaining 2.1 IDF1. This table compares association methods under the same detector, and BYTE's contribution is mainly in IDF1 and ID switches.

Leaderboards

BenchmarkMOTAIDF1HOTAID switchesFPS
MOT1780.377.363.1219629.6
MOT2077.875.261.3122317.5

Compiled from the ByteTrack rows of Tables 4 and 5 of the paper.

On the MOT17 test set, ByteTrack beats ReMOT, second place at the time, by 3.3 MOTA, 5.3 IDF1 and 3.4 HOTA, and is also the fastest in the whole table. The authors specifically mention that ByteTrack's training data is about 29,000 images, fewer than the 73,000 that many methods use.

MOT20 averages 170 people per test image, which is far more crowded. The paper says ByteTrack raises MOTA from 68.6 to 77.8 and cuts ID switches from 4209 to 1223, a 71% reduction. On HiEve it gets MOTA 61.7 and IDF1 63.1; on the BDD100K test set, mMOTA goes from 35.5 to 40.1. It ranks first on all four benchmarks.

Two things need to be kept apart here. A good part of ByteTrack's big lead on the leaderboards comes from YOLOX-X, a strong detector. BYTE's own contribution under the same detector is in the two sets of ablations above: against SORT, 2 points of MOTA and 2.4 points of IDF1; applied to other trackers, about 1 to 2 points of MOTA and up to nearly 10 points of IDF1. The appendix also compares smaller YOLOX models, and BYTE beats DeepSORT at every detector size, with MOTA 3 points higher on the smallest, YOLOX-Nano.

Afterward

ByteTrack quickly became the default choice in engineering. Ultralytics' tracking module offers two trackers, ByteTrack and BoT-SORT, and positions ByteTrack as the fastest and simplest baseline. OC-SORT and BoT-SORT both reuse ByteTrack's YOLOX weights directly for comparison, effectively treating it as the new generation's SORT.

Its weakness was spotted early, too. The DanceTrack paper reports a ByteTrack HOTA of only 47.7 and an AssA of 32.1. The dancers dress alike and move nonlinearly, Kalman filter plus IoU matching can't hold up, and both MOTR and OC-SORT beat it on that dataset. Connecting people by position alone works well when people walk in straight lines, and stops being enough when they spin.

Conclusion

Take the low-score boxes the threshold threw away and match them once more against the tracks left unmatched in the previous round, using IoU only. That single change is ByteTrack's whole method. SORT's ID switches drop by nearly half, nine different trackers almost all improve with it, and paired with YOLOX it takes first place on four benchmarks.

Next time tuning a tracker's detection threshold drives you up the wall, try BYTE first. It turns the threshold from a cutoff that decides which boxes live or die into something that only decides "who gets matched first."

Hegel probably never imagined that one day he'd be vouching for low-score boxes in a tracking paper.