Back to paper library
arXiv 2022WACV 2023Object trackingPaper notes

C-BIoU

A Box Around the Box

Original paper

Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Space

Paper authors

Fan Yang · Shigeyuki Odashima · Shoichi Masui · Shan Jiang

Publication
WACV 2023Conference
First submitted to arXiv
arXiv:2211.14317
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingcbiou2022,
  title = {{Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Space}},
  author = {Fan Yang and Shigeyuki Odashima and Shoichi Masui and Shan Jiang},
  year = {2022},
  eprint = {2211.14317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.14317}
}
On this page

After DanceTrack came out, people found that the same trackers that score about the same on MOT17 spread apart on DanceTrack. The dancers dress exactly alike, so appearance features are no help; and their moves are fast and erratic, so the Kalman filter often predicts wrong.

OC-SORT chose to fix the Kalman filter. Fan Yang, Shigeyuki Odashima, Shoichi Masui and Shan Jiang of Fujitsu Research took a different route, setting aside whether the motion model is accurate and widening the matching range a little.

The paper's title asks whether objects with irregular motion and similar appearance are hard to track, then answers by adding some buffer to the matching space. This method was also the main component of their second-place finishes in both the 2022 SoccerNet soccer tracking challenge and the DanceTrack group dance tracking challenge.

Defining the problem

When associating with box positions alone, trackers in the SORT family rely on IoU: the track from the previous frame predicts a box, the detection in this frame also has a box, and the more the two boxes overlap, the more likely they're the same person.

The authors suspect that tracking fails on DanceTrack mainly for two reasons:

  1. Adjacent frames don't overlap at all. One big move from a dancer, and the same person's boxes in two frames come apart. IoU is 0, and they can't be matched.
  2. Once the motion estimate is wrong, it stays wrong. When someone is hidden for a few frames, the track can only be pushed forward by the motion model. If they suddenly speed up or turn during that time, the predicted box drifts further and further off, so when they reappear, even with the detection box right there, the track can't match it.

Appearance features could normally rescue both situations, but that doesn't work when the dancers dress alike.

What the two problems have in common is that IoU gives every non-overlapping pair of boxes a score of 0. Missing by a hair and missing by a mile look the same to IoU.

IoU variants such as GIoU and DIoU can also compute a score for non-overlapping boxes. The authors propose a more direct approach.

Solving the problem

Stretch the box a little

Original Fig. 2. The top half is ordinary IoU: the overlap area of two boxes A and B divided by their union area. The bottom half is BIoU: A and B each get a buffer ring of green dashed lines around them, and overlap and union are computed with the enlarged boxes. The bottom explains that the original box and the buffered box share the same center, the original box has width and height w and h, the buffered box has width and height b_w and b_h, and the buffer ratio is b = (b_w − w) / 2w = (b_h − h) / 2h.
Original Fig. 2, Yang et al., from PDF page 2, arXiv license, scholarly quotation.

Buffered IoU (BIoU) simply stretches both boxes proportionally before computing IoU.

Write a box as its top-left corner coordinates plus width and height, o=(x,y,w,h)o = (x, y, w, h). With a buffer ratio of bb, the stretched box is

ob=(x−bw,  y−bh,  w+2bw,  h+2bh)o_b = (x - bw,\; y - bh,\; w + 2bw,\; h + 2bh)

It gains bwbw on the left and on the right, and bhbh on the top and on the bottom. The center stays put, the aspect ratio stays the same; the box just gets bigger.

Let's work through a concrete example. A dancer's box is 40 pixels wide and 100 tall, and in the next frame they jump 50 pixels to the right. The two boxes don't overlap at all, and IoU is 0.

  • b=0.3b = 0.3: Both boxes become 64 × 160. The centers are 50 apart and the width is 64, so they overlap by 64−50=1464 - 50 = 14 pixels horizontally; vertically they're perfectly aligned and overlap by 160 pixels. The intersection is 14×160=2,24014 \times 160 = 2{,}240, the union is 2×64×160−2,240=18,2402 \times 64 \times 160 - 2{,}240 = 18{,}240, and BIoU is about 0.12.
  • b=0.5b = 0.5: Both boxes become 80 × 200, overlapping by 30 pixels horizontally; the intersection is 6,000, the union 26,000, and BIoU is about 0.23.

Where the score used to be 0, there is now a score.

This score does two things. First, it lets the same person directly match across two adjacent frames even when the boxes don't overlap. Second, when the motion model's prediction is a bit off, the stretched box can still take in the real detection box, which amounts to compensating for motion estimation error in the matching space.

Unlike the old "search window" idea, the authors don't use the enlarged box to limit the search range; they use it directly to compute the matching score.

Too big won't work either

The bigger the buffer, the more people it can recover, but also the easier it is to match the wrong person.

Continue the example above. Suppose there's another dancer nearby, 70 pixels away. With b=0.3b = 0.3, the stretched box is 64 wide and can't reach someone 70 pixels away, so BIoU is 0; with b=0.5b = 0.5 it's 80 wide, overlaps them by 10 pixels, and BIoU is about 0.07. Enlarge the buffer and the neighbors become candidates too.

So the authors split matching into two stages:

Original Fig. 4. The C-BIoU tracker pipeline. Detections are first matched with alive tracks using BIoU with a small buffer (for example b1 = 0.3), and matched tracks update their state; unmatched tracks and unmatched detections are then matched once more using BIoU with a large buffer (for example b2 = 0.5). Together the two stages are called cascaded matching. Afterward, unmatched detections become new tracks; unmatched tracks have their age increased by one and are deleted once it exceeds max_age. State update: new tracks use the current box directly, and other tracks use the current state plus the average displacement over the last few frames.
Original Fig. 4, Yang et al., from PDF page 4, arXiv license, scholarly quotation.
  1. Match with a small buffer first. Alive tracks and this frame's detections are scored with BIoU using a small buffer b1b_1, then matched with the Hungarian algorithm. Most people moving normally are matched in this step.
  2. Match what's left once more with a large buffer. Tracks and detections left unmatched in the first round are matched again using a large buffer b2b_2. This round handles people with especially big moves, or whose motion estimate has drifted further.

This kind of cascaded matching, "match the confident ones first, then the hard ones," is very common in tracking. ByteTrack matches high-confidence detections first, and DeepSORT matches the most recently matched tracks first; C-BIoU first uses a more conservative matching space, and only widens it if anything is left over.

How are the two buffer ratios set? The authors run a grid search on the training set. Since b1<b2b_1 < b_2, the number of combinations is limited, and since this tracker runs very fast, the time spent searching is acceptable.

No Kalman filter

C-BIoU's motion model is surprisingly simple, just averaging the displacement over the last few frames and pushing forward.

If a track has been matched for more than nn consecutive frames and then goes Δ\Delta frames unmatched, its state is estimated as

st+Δ=st+Δn∑i=t−n+1t(oi−oi−1)s_{t+\Delta} = s_t + \frac{\Delta}{n}\sum_{i=t-n+1}^{t}\left(o^{i} - o^{i-1}\right)

Here a box is written as its top-left and bottom-right corners, o=(x,y,x+w,y+h)o = (x, y, x + w, y + h). nn defaults to somewhere between 2 and 5.

The total displacement over the last nn frames divided by nn is the average movement per frame, which is then multiplied by the number of frames in the gap. When a dancer suddenly changes direction, this estimate catches up within two or three frames at most, rather than waiting for the velocity to be corrected bit by bit as with a Kalman filter. Whatever the estimate gets wrong is left to the buffer to make up for.

Track management follows SORT's approach. All detections in the first frame become new tracks; a new track has no motion to estimate yet, so the recorded box is used directly as its current state. In each frame, a matched track's age resets to zero and an unmatched one's goes up by one, and it's deleted once it exceeds max_age; detections still left after both rounds of matching become new tracks.

The authors also explain that this is just a simple prototype demonstrating how to use C-BIoU, and other modules can be added if needed.

Discussion

Same detections, MOT17 and DanceTrack

The authors first compare on the MOT17 and DanceTrack test sets. For fairness, the methods below all use the same YOLOX-X detections (the MOT17 weights come from ByteTrack, and the DanceTrack ones from OC-SORT):

MethodMOT17 HOTADanceTrack HOTADanceTrack AssADanceTrack IDF1
DeepSORT61.245.629.747.9
SORT63.050.033.252.0
ByteTrack63.151.933.852.0
OC-SORT63.255.738.354.6
C-BIoU64.160.645.461.6

Compiled from Table I of the paper, test sets, same detections.

On MOT17, the five methods are only 3 points apart in HOTA. The authors cite the DanceTrack paper's analysis that on MOT17 the bottleneck is detection, not tracking.

On DanceTrack, they spread apart. C-BIoU beats OC-SORT by 4.9 points of HOTA, AssA, which measures association, goes from 38.3 to 45.4, and IDF1 from 54.6 to 61.6. The two have about the same detection DetA (81.3 vs. 81.7), so the gap comes almost entirely from association. The top half of the paper's table lists methods that use other detectors, for reference only; among them, the end-to-end MOTR is at 54.2.

Tracking only: ground-truth boxes

Detection quality can muddy the comparison, so the authors also feed ground-truth boxes directly as detections and compare association alone. On the DanceTrack validation set, SoccerNet soccer videos and GMOT-40, C-BIoU's HOTA is 81.7, 89.2 and 96.4, while the runner-up, OC-SORT, gets 79.1, 82.0 and 92.4. The gaps range from 2.6 to 7.2 points, with the largest on SoccerNet.

GMOT-40 is a multi-object tracking dataset of generic objects, where the frame often holds a big group of identical objects of the same kind; SoccerNet is soccer broadcast footage, where the camera follows the ball.

The authors also note that even with ground-truth boxes, the results on all three datasets are still not perfect. Building benchmarks on ground-truth boxes and focusing on improving association still makes sense for now.

What each part contributes

The ablations use ground-truth boxes. They first remove both cascaded matching and motion estimation, and compare only which similarity to use in a single round of matching:

SimilarityDanceTrackSoccerNetGMOT-40
IoU76.681.993.0
GIoU77.179.893.4
DIoU75.184.393.6
BIoU80.087.796.2
+ cascaded matching80.288.996.3
+ motion estimation81.789.296.4

Compiled from Table IV of the paper, HOTA; DanceTrack is the validation set and the other two are test sets, all with ground-truth boxes as detections.

  1. BIoU is the main contribution. With the same single round of matching and no motion estimation, BIoU beats IoU by 3 to 6 points on all three datasets. GIoU and DIoU can also handle non-overlapping boxes, but don't improve things consistently, and DIoU even scores lower than IoU on DanceTrack.
  2. Cascaded matching helps most on SoccerNet. Adding cascaded matching gains only 0.2 and 0.1 points on DanceTrack and GMOT-40, but 1.2 points on SoccerNet. The authors' explanation is that the broadcast camera moves, making the players' motion extremely fast, and that's when the large buffer in the second round comes into play.
  3. The average displacement adds 1.5 points on DanceTrack. Simple as it is, having motion estimation still beats having none, because the buffer can compensate for the estimation error.

Buffer ratios

Original Fig. 6. HOTA on the DanceTrack validation set for different combinations of buffer ratios. The horizontal axis is the small buffer b1 and the vertical axis the large buffer b2, both from 0.1 to 0.7, taking only the lower triangle where b1 is smaller than b2. The highest is at b1 = 0.3, b2 = 0.4, with 81.7; the whole b1 = 0.3 column lies between 81.4 and 81.7; the lowest is at b1 = 0.6, b2 = 0.7, with 78.2.
Original Fig. 6, Yang et al., from PDF page 7, arXiv license, scholarly quotation.

Read this figure column by column. Within one column, b1b_1 is fixed and b2b_2 varies, and the score barely moves, with differences all within 0.3 points; move to another column and the score changes noticeably. In other words, the result is mainly decided by the small buffer in the first round, with b1=0.3b_1 = 0.3 best and both larger and smaller values doing worse.

The worst combination is 78.2, still higher than IoU's 76.6. The buffer ratio does affect the result, but any buffer at all beats none.

When detection is poor

The authors found this method's limitation through their own experiments. They add noise to the ground-truth boxes of the DanceTrack validation set, first deleting a proportion of the boxes to create misses, then adding the same proportion of fake boxes where there are no people to create false positives.

NoiseOC-SORTIoUC-BIoU
0%79.176.681.7
20%61.457.662.3
40%28.038.329.2

Compiled from Table V of the paper, DanceTrack validation set, HOTA.

With noise at 20% or less, C-BIoU is still the best. At 40%, it drops to 29.2, actually 9 points below the simplest IoU tracker (38.3); its AssA, which measures association, is down to just 14.7, while the IoU tracker still has 25.0.

The authors explain that when there's little noise, a person's track and detections fail to line up only for an occasional frame or two, and the buffer can link them back together. When there's a lot of noise, a person may have no detections for many consecutive frames, and neither IoU nor BIoU can link them back; worse, the enlarged matching space matches tracks to nearby fake boxes more aggressively.

The authors argue that on the datasets they care about, high-quality detections are usually available, because everyone looking alike actually makes detection easier. That's the authors' judgment of their own scope; in scenes where the detector is unreliable, check detection quality first.

Speed

Dropping the Kalman filter also makes the tracking itself much faster. On an Intel Xeon Silver 4216 CPU, excluding detection time, C-BIoU processes 361 frames per second on MOT17 and 680 on DanceTrack; OC-SORT does 185 and 341, and SORT 144 and 271. The authors also note that the more objects there are, the slower the tracker gets.

Conclusion

C-BIoU's method takes one sentence to describe. It stretches the boxes before computing IoU. Add a two-stage matching and a motion model that averages the displacement over the last few frames, and it beats OC-SORT on DanceTrack by nearly 5 points of HOTA, while running twice as fast.

Judging by the ablations, most of the credit goes to BIoU itself, with the first-round buffer ratio the most critical; cascaded matching is clearly useful only on soccer videos where the camera moves. The price is that once detection gets very poor, the enlarged matching space makes the tracker more likely to match fake boxes.

The paper doesn't come with code, but the idea is simple enough to implement yourself. A 2025 survey of multi-object tracking lists C-BIoU as the representative of the "expand the box before association" approach, and points out that the later Deep-EIoU generalized the fixed two-stage buffer into multiple stages that grow step by step.

"Buffer" was a well-chosen word. When the motion model can't be sure, just leave a little room.