Iterative Scale-Up ExpansionIoU and Deep Features Association for Multi-Object Tracking in Sports
- Publication
- WACV Workshops 2024Workshop
- First submitted to arXiv
- arXiv:2306.13074
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingdeepeiou2023,
title = {{Iterative Scale-Up ExpansionIoU and Deep Features Association for Multi-Object Tracking in Sports}},
author = {Hsiang-Wei Huang and Cheng-Yen Yang and Jiacheng Sun and Pyong-Kun Kim and Kwang-Ju Kim and Kyoungoh Lee and Chung-I Huang and Jenq-Neng Hwang},
year = {2023},
eprint = {2306.13074},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2306.13074}
}
The SportsMOT paper has one ablation table that's hard to shake. With IoU association in both cases, the version that first predicts positions with a Kalman filter scores 64.1 HOTA; skipping prediction and comparing directly against the track's box from the previous frame actually scores 71.5.
If prediction only gets in the way, why predict at all?
Hsiang-Wei Huang, Cheng-Yen Yang and Jiacheng Sun of Jenq-Neng Hwang's lab at the University of Washington, together with collaborators from Korea's Electronics and Telecommunications Research Institute (ETRI) and Taiwan's National Center for High-Performance Computing, proposed Deep-EIoU. It removes the Kalman filter entirely, uses a heavily stretched box to find people instead, and adds appearance features to tell who's who.
This method reaches 77.2 HOTA on SportsMOT, 3 points above the best method in the SportsMOT paper.
Defining the problem
The authors list three reasons sports scenes are hard to track:
- Heavy occlusion. Competition is intense, and players often bunch up and block one another, far more than in pedestrian tracking.
- Similar appearance. Players on the same team wear jerseys of the same color.
- Unpredictable motion. In a between-the-legs dribble in basketball, a slide tackle in soccer or a spike in volleyball, speed and direction change in an instant.
Mainstream trackers, whether they fuse appearance and motion or rely on motion alone, mostly predict positions with a Kalman filter. The Kalman filter assumes linear motion and Gaussian noise, and once an object's motion doesn't fit, the prediction drifts. OC-SORT tries to patch the Kalman filter; C-BIoU replaces it with the average displacement and stretches the box to make up for estimation error.
Deep-EIoU goes one step further and doesn't do motion estimation at all.
Solving the problem

The overall structure is still detect-then-associate, using only past and present information, so it can run online. In each frame, YOLOX first detects the players, a ReID model extracts appearance features for each box, and then the costs between tracks and detections are computed and handed to the Hungarian algorithm for matching.
The biggest difference from ordinary trackers is the box a track uses for comparison. Deep-EIoU doesn't predict where a track will be in this frame; it takes the track's box from the last time it was seen and compares it directly with the new detections.
A box stretched very wide
For position, it uses ExpansionIoU (EIoU). The authors explain that the idea comes from C-BIoU, but they found that for athletes, the box needs to be stretched even wider.
The expansion is controlled by a scale . For a box of width and height , the expansion lengths are
These lengths are split in half and added to both sides of the box, so the top-left corner moves outward by , and so does the bottom-right corner. So the expanded box is times the original in both width and height. The official code is written consistently with this definition.
With the default , the box's width and height become 3.4 times the original, and its area nearly 12 times. In C-BIoU's notation, that's equivalent to a buffer ratio of 1.2; C-BIoU's best buffer on DanceTrack is only 0.3 to 0.4.
Let's reuse the example from the C-BIoU note for comparison. A player 40 pixels wide and 100 tall moves 50 pixels to the right in the next frame, and the original IoU is 0:
| Expanded box | Same person (50 px) | Neighbor (70 px) | Neighbor (100 px) | |
|---|---|---|---|---|
| C-BIoU () | 64 × 160 | 0.12 | 0 | 0 |
| Deep-EIoU () | 136 × 340 | 0.46 | 0.32 | 0.15 |
Worked example for this note; both boxes are the same size and vertically aligned.
Expanded this much, a player can still be found even after moving more than a body width in an instant. The price is just as plain to see, with everyone within three body widths on either side turning into a candidate. That's when appearance has to separate them.
Combining appearance and position
The appearance cost is the cosine distance between the feature vectors of a track and a detection:
Many methods add the appearance cost and the IoU cost with weights. The authors don't; they adopt BoT-SORT's approach instead. First the appearance cost is adjusted. If or , the adjusted appearance cost is ; otherwise . The first-stage cost then takes the smaller of the two:
Here and .
For appearance to come into play, the two have to look similar enough and also be close enough in position (an EIoU of at least 0.5). If either condition fails, the appearance cost is set to 1, which amounts to ignoring appearance; if both hold, the appearance cost is then cut in half. Finally, the smaller of the appearance and position costs is taken.
So appearance can only make pairs that are already close "more certain"; it can't force together two people who are far apart and merely look alike. That matters especially when teammates all wear the same jersey.
Confident matches first
Matching follows ByteTrack's tiers. High-score detections, with a confidence of at least 0.6, are usually less occluded and have more reliable appearance features, so they're matched with tracks first, using the combined appearance and EIoU cost above; low-score detections, with confidence between 0.1 and 0.6, are matched in the second round against the remaining tracks using EIoU only; anything below 0.1 is thrown away.
In addition, the authors remove the aspect-ratio constraint that ordinary trackers place on boxes, because a player might be lying on the ground, unlike pedestrians, who are mostly upright and walking. Lost tracks are kept for 60 frames, longer than the 30 frames common in pedestrian tracking, because occlusion on the court is more severe.
Scale up step by step, no tuning needed
C-BIoU's ablations already showed that the buffer ratio is a sensitive hyperparameter. The authors point out that it's hard to tune in a real athlete tracking system, because the system has to run online in real time, with no chance for repeated experiments after the fact, and in real-world scenes there's no ground truth for the test data, so there's nothing to search against anyway.
So instead of looking for the best , they increase step by step from small to large during the matching stage for high-score detections, repeating the matching:
starts at 0. The first round uses and settles the pairs that overlap most and are most certain; whatever is left unmatched gets searched again in the next round with a somewhat bigger box. The defaults are and a step of , for two rounds in total, that is, 0.7 and 0.8.
Discussion
SportsMOT
The detector is YOLOX-X, starting from COCO-pretrained weights and fine-tuned for 80 epochs on the SportsMOT training and validation sets. The ReID model is OSNet, starting from weights trained on the person re-identification dataset Market-1501 and then fine-tuned on 31,279 player images cropped from SportsMOT ground-truth boxes.
| Method | Training data | HOTA | AssA | DetA | IDF1 |
|---|---|---|---|---|---|
| BoT-SORT | Train | 68.7 | 55.9 | 84.4 | 70.0 |
| OC-SORT | Train | 71.9 | 59.8 | 86.4 | 72.2 |
| Deep-EIoU | Train | 74.1 | 63.1 | 87.2 | 75.0 |
| OC-SORT | Train + val | 73.7 | 61.5 | 88.5 | 74.0 |
| MixSort-OC | Train + val | 74.1 | 62.0 | 88.5 | 74.4 |
| Deep-EIoU | Train + val | 77.2 | 67.7 | 88.2 | 79.8 |
Compiled from Table 2 of the paper, SportsMOT test set. BoT-SORT was rerun with its official code; the other baseline numbers are taken from the SportsMOT paper.
Compared on the same training data, Deep-EIoU is the best under both settings. With training plus validation data, its HOTA is 3.1 points above MixSort-OC, and association AssA is 5.7 points higher; detection DetA is about the same, even slightly lower. Almost the entire gap comes from association.
SoccerNet
SoccerNet-Tracking has 201 soccer videos, 30 seconds each, and what has to be tracked isn't only the players but also goalkeepers, referees and the ball. To compare tracking alone, all methods use the ground-truth boxes provided by the dataset directly.
Deep-EIoU's HOTA is 85.4; the runner-up, OC-SORT, gets 78.1, BoT-SORT 77.0, and ByteTrack and DeepSORT only 71.5 and 69.6.
What each part contributes
| Setting | HOTA |
|---|---|
| Baseline (EIoU only) | 71.4 |
| + appearance (ReID) | 75.3 |
| + iterative scale-up | 77.2 |
| + linear interpolation | 77.2 |
Compiled from Table 4 of the paper, SportsMOT test set.
- Appearance contributes the most, 3.9 points. Teammates may look very alike, but appearance is still an important cue. This matches the SportsMOT paper's conclusion of "similar yet distinguishable."
- Iterative scale-up adds another 1.9 points. The authors also explain that this step simultaneously extends how long lost tracks are kept from 30 frames to 60, so the 1.9 points are the combined effect of both.
- Linear interpolation has almost no effect, adding less than 0.1 points.
The baseline with EIoU only and no appearance is 71.4, very close to the SportsMOT paper's "IoU without Kalman" row (71.5). The two papers train their detectors differently, so this is only a rough comparison; judging by these numbers, stretching the box on its own isn't much better than IoU without prediction, and Deep-EIoU's extra points come mainly from appearance and iterative scale-up.
Does the expansion scale really need no tuning?

The authors vary the initial expansion scale from 0.2 to 0.8, keeping everything else unchanged. HOTA stays between 76.3 and 77.2, all above the previous best of 74.1. From this the authors conclude that iterative scale-up frees the method from parameter tuning.
This figure can also be read from another angle. With an initial scale of 0.2, the two rounds expand by 0.2 and 0.3, enlarging the box's width and height to 2.4 to 2.6 times; at 0.7, it's 3.4 to 3.6 times, and HOTA gains only 0.9 points. Within this range, how much you expand doesn't actually matter much.
Still, even the smallest expansion in the figure is 2.4 times, already bigger than C-BIoU's setting. The authors say "expanding more than C-BIoU is clearly better," but this figure doesn't directly test that; judging by the ablation table, what really sets Deep-EIoU apart is still appearance and iterative scale-up.
EIoU can help the Kalman filter too
Finally, the authors drop EIoU straight into two Kalman filter-based trackers, expanding both the predicted boxes and the detection boxes before computing IoU. ByteTrack's HOTA rises from 62.8 to 67.5, and BoT-SORT's from 68.7 to 71.3.
So EIoU isn't limited to trackers that don't predict; it also works as a small plug-and-play trick. But even with EIoU added, these two Kalman trackers still lose to Deep-EIoU, which predicts nothing at all.
Limitations
The authors list two limitations in the paper:
- No offline trajectory refinement. When a player leaves the frame and comes back, an online tracker has a hard time recognizing them again. The authors mention that post-processing or a stronger memory mechanism could be added, but this paper doesn't do it.
- Speed. With the detector and the ReID model included, the whole pipeline runs at about 14.6 frames per second on one RTX 4080, slower than motion-only trackers; switching to a lighter detector and ReID model could speed it up.
Conclusion
Deep-EIoU makes a bold choice for sports scenes. It doesn't predict where a player will be in the next frame; it takes only the last position where the player was seen, stretches the box to 3.4 times its size to look for them, and then uses appearance to filter out the candidates that flood in once the box is stretched. Appearance can only make pairs that are already close enough more certain, not pull in a lookalike from far away; and the expansion scale is tried in two rounds from small to large, which saves parameter tuning.
The ablations show that among the three designs, appearance contributes the most, followed by iterative scale-up. The EIoU-only baseline isn't far from the SportsMOT paper's IoU without prediction.
A 2025 survey of multi-object tracking describes Deep-EIoU as a generalization of C-BIoU, going from a fixed two-stage expansion to iterative scale-up, and ranks it second in its SportsMOT comparison table, behind only the survey authors' own method. The survey also notes that Deep-EIoU has only been evaluated on SportsMOT and SoccerNet.
As for the first limitation the paper itself admits, not recognizing players who leave the frame and come back, the same lab answered it with GTA a little over a year later.