Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Space
- Publication
- WACV 2023Conference
- First submitted to arXiv
- arXiv:2211.14317
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingcbiou2022,
title = {{Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Space}},
author = {Fan Yang and Shigeyuki Odashima and Shoichi Masui and Shan Jiang},
year = {2022},
eprint = {2211.14317},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2211.14317}
}
After DanceTrack came out, people found that the same trackers that score about the same on MOT17 spread apart on DanceTrack. The dancers dress exactly alike, so appearance features are no help; and their moves are fast and erratic, so the Kalman filter often predicts wrong.
OC-SORT chose to fix the Kalman filter. Fan Yang, Shigeyuki Odashima, Shoichi Masui and Shan Jiang of Fujitsu Research took a different route, setting aside whether the motion model is accurate and widening the matching range a little.
The paper's title asks whether objects with irregular motion and similar appearance are hard to track, then answers by adding some buffer to the matching space. This method was also the main component of their second-place finishes in both the 2022 SoccerNet soccer tracking challenge and the DanceTrack group dance tracking challenge.
Defining the problem
When associating with box positions alone, trackers in the SORT family rely on IoU: the track from the previous frame predicts a box, the detection in this frame also has a box, and the more the two boxes overlap, the more likely they're the same person.
The authors suspect that tracking fails on DanceTrack mainly for two reasons:
- Adjacent frames don't overlap at all. One big move from a dancer, and the same person's boxes in two frames come apart. IoU is 0, and they can't be matched.
- Once the motion estimate is wrong, it stays wrong. When someone is hidden for a few frames, the track can only be pushed forward by the motion model. If they suddenly speed up or turn during that time, the predicted box drifts further and further off, so when they reappear, even with the detection box right there, the track can't match it.
Appearance features could normally rescue both situations, but that doesn't work when the dancers dress alike.
What the two problems have in common is that IoU gives every non-overlapping pair of boxes a score of 0. Missing by a hair and missing by a mile look the same to IoU.
IoU variants such as GIoU and DIoU can also compute a score for non-overlapping boxes. The authors propose a more direct approach.
Solving the problem
Stretch the box a little

Buffered IoU (BIoU) simply stretches both boxes proportionally before computing IoU.
Write a box as its top-left corner coordinates plus width and height, . With a buffer ratio of , the stretched box is
It gains on the left and on the right, and on the top and on the bottom. The center stays put, the aspect ratio stays the same; the box just gets bigger.
Let's work through a concrete example. A dancer's box is 40 pixels wide and 100 tall, and in the next frame they jump 50 pixels to the right. The two boxes don't overlap at all, and IoU is 0.
- : Both boxes become 64 × 160. The centers are 50 apart and the width is 64, so they overlap by pixels horizontally; vertically they're perfectly aligned and overlap by 160 pixels. The intersection is , the union is , and BIoU is about 0.12.
- : Both boxes become 80 × 200, overlapping by 30 pixels horizontally; the intersection is 6,000, the union 26,000, and BIoU is about 0.23.
Where the score used to be 0, there is now a score.
This score does two things. First, it lets the same person directly match across two adjacent frames even when the boxes don't overlap. Second, when the motion model's prediction is a bit off, the stretched box can still take in the real detection box, which amounts to compensating for motion estimation error in the matching space.
Unlike the old "search window" idea, the authors don't use the enlarged box to limit the search range; they use it directly to compute the matching score.
Too big won't work either
The bigger the buffer, the more people it can recover, but also the easier it is to match the wrong person.
Continue the example above. Suppose there's another dancer nearby, 70 pixels away. With , the stretched box is 64 wide and can't reach someone 70 pixels away, so BIoU is 0; with it's 80 wide, overlaps them by 10 pixels, and BIoU is about 0.07. Enlarge the buffer and the neighbors become candidates too.
So the authors split matching into two stages:

- Match with a small buffer first. Alive tracks and this frame's detections are scored with BIoU using a small buffer , then matched with the Hungarian algorithm. Most people moving normally are matched in this step.
- Match what's left once more with a large buffer. Tracks and detections left unmatched in the first round are matched again using a large buffer . This round handles people with especially big moves, or whose motion estimate has drifted further.
This kind of cascaded matching, "match the confident ones first, then the hard ones," is very common in tracking. ByteTrack matches high-confidence detections first, and DeepSORT matches the most recently matched tracks first; C-BIoU first uses a more conservative matching space, and only widens it if anything is left over.
How are the two buffer ratios set? The authors run a grid search on the training set. Since , the number of combinations is limited, and since this tracker runs very fast, the time spent searching is acceptable.
No Kalman filter
C-BIoU's motion model is surprisingly simple, just averaging the displacement over the last few frames and pushing forward.
If a track has been matched for more than consecutive frames and then goes frames unmatched, its state is estimated as
Here a box is written as its top-left and bottom-right corners, . defaults to somewhere between 2 and 5.
The total displacement over the last frames divided by is the average movement per frame, which is then multiplied by the number of frames in the gap. When a dancer suddenly changes direction, this estimate catches up within two or three frames at most, rather than waiting for the velocity to be corrected bit by bit as with a Kalman filter. Whatever the estimate gets wrong is left to the buffer to make up for.
Track management follows SORT's approach. All detections in the first frame become new tracks; a new track has no motion to estimate yet, so the recorded box is used directly as its current state. In each frame, a matched track's age resets to zero and an unmatched one's goes up by one, and it's deleted once it exceeds max_age; detections still left after both rounds of matching become new tracks.
The authors also explain that this is just a simple prototype demonstrating how to use C-BIoU, and other modules can be added if needed.
Discussion
Same detections, MOT17 and DanceTrack
The authors first compare on the MOT17 and DanceTrack test sets. For fairness, the methods below all use the same YOLOX-X detections (the MOT17 weights come from ByteTrack, and the DanceTrack ones from OC-SORT):
| Method | MOT17 HOTA | DanceTrack HOTA | DanceTrack AssA | DanceTrack IDF1 |
|---|---|---|---|---|
| DeepSORT | 61.2 | 45.6 | 29.7 | 47.9 |
| SORT | 63.0 | 50.0 | 33.2 | 52.0 |
| ByteTrack | 63.1 | 51.9 | 33.8 | 52.0 |
| OC-SORT | 63.2 | 55.7 | 38.3 | 54.6 |
| C-BIoU | 64.1 | 60.6 | 45.4 | 61.6 |
Compiled from Table I of the paper, test sets, same detections.
On MOT17, the five methods are only 3 points apart in HOTA. The authors cite the DanceTrack paper's analysis that on MOT17 the bottleneck is detection, not tracking.
On DanceTrack, they spread apart. C-BIoU beats OC-SORT by 4.9 points of HOTA, AssA, which measures association, goes from 38.3 to 45.4, and IDF1 from 54.6 to 61.6. The two have about the same detection DetA (81.3 vs. 81.7), so the gap comes almost entirely from association. The top half of the paper's table lists methods that use other detectors, for reference only; among them, the end-to-end MOTR is at 54.2.
Tracking only: ground-truth boxes
Detection quality can muddy the comparison, so the authors also feed ground-truth boxes directly as detections and compare association alone. On the DanceTrack validation set, SoccerNet soccer videos and GMOT-40, C-BIoU's HOTA is 81.7, 89.2 and 96.4, while the runner-up, OC-SORT, gets 79.1, 82.0 and 92.4. The gaps range from 2.6 to 7.2 points, with the largest on SoccerNet.
GMOT-40 is a multi-object tracking dataset of generic objects, where the frame often holds a big group of identical objects of the same kind; SoccerNet is soccer broadcast footage, where the camera follows the ball.
The authors also note that even with ground-truth boxes, the results on all three datasets are still not perfect. Building benchmarks on ground-truth boxes and focusing on improving association still makes sense for now.
What each part contributes
The ablations use ground-truth boxes. They first remove both cascaded matching and motion estimation, and compare only which similarity to use in a single round of matching:
| Similarity | DanceTrack | SoccerNet | GMOT-40 |
|---|---|---|---|
| IoU | 76.6 | 81.9 | 93.0 |
| GIoU | 77.1 | 79.8 | 93.4 |
| DIoU | 75.1 | 84.3 | 93.6 |
| BIoU | 80.0 | 87.7 | 96.2 |
| + cascaded matching | 80.2 | 88.9 | 96.3 |
| + motion estimation | 81.7 | 89.2 | 96.4 |
Compiled from Table IV of the paper, HOTA; DanceTrack is the validation set and the other two are test sets, all with ground-truth boxes as detections.
- BIoU is the main contribution. With the same single round of matching and no motion estimation, BIoU beats IoU by 3 to 6 points on all three datasets. GIoU and DIoU can also handle non-overlapping boxes, but don't improve things consistently, and DIoU even scores lower than IoU on DanceTrack.
- Cascaded matching helps most on SoccerNet. Adding cascaded matching gains only 0.2 and 0.1 points on DanceTrack and GMOT-40, but 1.2 points on SoccerNet. The authors' explanation is that the broadcast camera moves, making the players' motion extremely fast, and that's when the large buffer in the second round comes into play.
- The average displacement adds 1.5 points on DanceTrack. Simple as it is, having motion estimation still beats having none, because the buffer can compensate for the estimation error.
Buffer ratios

Read this figure column by column. Within one column, is fixed and varies, and the score barely moves, with differences all within 0.3 points; move to another column and the score changes noticeably. In other words, the result is mainly decided by the small buffer in the first round, with best and both larger and smaller values doing worse.
The worst combination is 78.2, still higher than IoU's 76.6. The buffer ratio does affect the result, but any buffer at all beats none.
When detection is poor
The authors found this method's limitation through their own experiments. They add noise to the ground-truth boxes of the DanceTrack validation set, first deleting a proportion of the boxes to create misses, then adding the same proportion of fake boxes where there are no people to create false positives.
| Noise | OC-SORT | IoU | C-BIoU |
|---|---|---|---|
| 0% | 79.1 | 76.6 | 81.7 |
| 20% | 61.4 | 57.6 | 62.3 |
| 40% | 28.0 | 38.3 | 29.2 |
Compiled from Table V of the paper, DanceTrack validation set, HOTA.
With noise at 20% or less, C-BIoU is still the best. At 40%, it drops to 29.2, actually 9 points below the simplest IoU tracker (38.3); its AssA, which measures association, is down to just 14.7, while the IoU tracker still has 25.0.
The authors explain that when there's little noise, a person's track and detections fail to line up only for an occasional frame or two, and the buffer can link them back together. When there's a lot of noise, a person may have no detections for many consecutive frames, and neither IoU nor BIoU can link them back; worse, the enlarged matching space matches tracks to nearby fake boxes more aggressively.
The authors argue that on the datasets they care about, high-quality detections are usually available, because everyone looking alike actually makes detection easier. That's the authors' judgment of their own scope; in scenes where the detector is unreliable, check detection quality first.
Speed
Dropping the Kalman filter also makes the tracking itself much faster. On an Intel Xeon Silver 4216 CPU, excluding detection time, C-BIoU processes 361 frames per second on MOT17 and 680 on DanceTrack; OC-SORT does 185 and 341, and SORT 144 and 271. The authors also note that the more objects there are, the slower the tracker gets.
Conclusion
C-BIoU's method takes one sentence to describe. It stretches the boxes before computing IoU. Add a two-stage matching and a motion model that averages the displacement over the last few frames, and it beats OC-SORT on DanceTrack by nearly 5 points of HOTA, while running twice as fast.
Judging by the ablations, most of the credit goes to BIoU itself, with the first-round buffer ratio the most critical; cascaded matching is clearly useful only on soccer videos where the camera moves. The price is that once detection gets very poor, the enlarged matching space makes the tracker more likely to match fake boxes.
The paper doesn't come with code, but the idea is simple enough to implement yourself. A 2025 survey of multi-object tracking lists C-BIoU as the representative of the "expand the box before association" approach, and points out that the later Deep-EIoU generalized the fixed two-stage buffer into multiple stages that grow step by step.
"Buffer" was a well-chosen word. When the motion model can't be sure, just leave a little room.