Back to paper library
CVPR 2025Object trackingPaper notes

TrackTrack

It's Mutual

Original paper

Focusing on Tracks for Online Multi-Object Tracking

Paper authors

Kyujin Shim · Kangwook Ko · Yujin Yang · Changick Kim

Publication
CVPR 2025Conference
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@inproceedings{multiobjecttrackingtracktrack2025,
  title = {{Focusing on Tracks for Online Multi-Object Tracking}},
  author = {Kyujin Shim and Kangwook Ko and Yujin Yang and Changick Kim},
  year = {2025},
  booktitle = {CVPR},
  url = {https://doi.org/10.1109/CVPR52734.2025.01091},
  doi = {10.1109/CVPR52734.2025.01091}
}
On this page

The Hungarian algorithm is the most common component in a tracker. Given a cost matrix of tracks against detections, it finds the one-to-one matching with the lowest total cost, and ever since SORT, almost every detect-then-associate tracker has used it.

The algorithm was originally designed for assignment problems, such as assigning workers to jobs so that the total cost is lowest. Any worker can take any job; what matters is that the whole works out cheapest.

Kyujin Shim, Changick Kim and colleagues at KAIST argue that tracking isn't that kind of problem. Tracks and detection boxes represent the same group of people, and each track has exactly one correct detection box in this frame. Chasing the lowest total cost can sometimes swap two people's identities just to save a tiny bit overall.

Defining the problem

Off by 0.01, identities swapped

Original Figure 2. An occlusion example from DanceTrack. Two tracks, T2 and T3, overlap, and beside them is the cost matrix of four tracks against four detection boxes. (a) The Hungarian algorithm picks T2–D4 and T3–D3 (red boxes on yellow), and in the next frame T2 and T3 swap identities, an ID switch. (b) TPA picks T2–D3 and T3–D4 (blue boxes on yellow), and both people are tracked correctly. D4 is a high-score box removed by NMS (yellow).
Original Figure 2, Shim et al., from PDF page 2, scholarly quotation.

T1 with D2 (0.16) and T4 with D1 (0.18) are uncontroversial; the problem lies in the four cells for T2, T3 and D3, D4:

AssignmentT2T3Total (4 pairs)
HungarianD4: 0.48D3: 0.331.15
TPAD3: 0.10D4: 0.721.16

Numbers from Figure 2 of the paper; totals computed for this note.

The two assignments differ in total cost by only 0.01. The Hungarian algorithm picks the one with the smaller sum, yet in that assignment T2, whose distance to D3 is only 0.10, the smallest cell in the whole table, gets sent off to D4 at 0.48.

D3 is most like T2, and T2 is most like D3. The attraction is mutual, so this pair is almost impossible to get wrong. Force them apart to save 0.01 overall, and the result is swapped identities.

Low-score boxes wait their turn

The other problem comes from the multi-stage matching used since ByteTrack. High-score boxes are matched to tracks first, and only the tracks left over get a turn with the low-score boxes.

But people who are occluded often get low detection scores. For some track, the most suitable box may be exactly a low-score box, yet in the first round the track has already been grabbed by a less suitable high-score box.

The more extreme case is boxes removed by NMS. When two people overlap heavily, NMS treats one person's box as a duplicate and removes it. D4 in the figure is such a box: its score is high, it just overlaps too much with someone else. Most trackers never give it a second look.

Solving the problem

TrackTrack has two strategies: TPA (track-perspective-based association) at matching time, and TAI (track-aware initialization) when creating new tracks. The whole method needs no extra training.

TPA: all boxes together, matched only when it's mutual

Original Figure 3. The TPA procedure. On the left is the cost matrix of five tracks against six detection boxes; green are high-score boxes, blue are low-score boxes, and yellow are high-score boxes removed by NMS. The first round finds the cells that are the minimum of both their row and their column (red boxes): T1–D6, T2–D2 and T4–D3. Removing those rows and columns leaves the small matrix on the right, where the second round finds T3–D1.
Original Figure 3, Shim et al., from PDF page 4, scholarly quotation.

TPA first puts three kinds of boxes into the same cost matrix: high-score boxes after NMS, low-score boxes after NMS, and high-score boxes removed by NMS. The latter two each get a penalty added to their cost, so that, other things being equal, they rank behind the high-score boxes without losing their chance entirely.

The cost itself follows Hybrid-SORT's combination: box overlap, cosine distance between appearance features, detection confidence, and the angle difference in motion direction.

Then it repeatedly finds every "mutual minimum" pair, that is, a cell that is the smallest in its row and also the smallest in its column, and below a threshold. All the pairs found are fixed, their rows and columns are removed, and it searches again in the remaining matrix. With each extra round, the threshold tightens a little.

Walking through the matrix in the figure:

  1. First round. The smallest in T1's row is D6 (0.1), and the smallest in D6's column is also T1, so they're matched. T2–D2 and T4–D3 likewise. The smallest in T3's row is D2 (0.2), but the smallest in D2's column is T2 (0.1), so T3 goes unmatched this round.
  2. Second round. T3, T5 and D1, D4, D5 remain. The smallest in T3's row is D1 (0.3), and in D1's column T3 is also smaller than T5's 0.9, so they're matched.

The process is a bit like speed dating. Two people who each think the other is the best choice in the room pair up first; those whose prospects are less clear wait until the sure matches are settled, then choose from whoever is left.

Handling the surest pairs first means never doing what the Hungarian algorithm does: breaking up the surest pair for the sake of the whole.

TAI: keep new tracks from colliding with old ones

After matching, the high-score boxes left over are usually treated as newly appeared people and given new tracks. The trouble is that some of these leftover boxes are actually duplicate detections of existing tracks, or extra boxes on someone who is occluded. Once turned into new tracks, they become a pile of short-lived fake tracks.

Original Figure 4. An illustration of TAI. On the left, the blue, green and purple boxes are the matched tracks T1, T2 and T3, and the red dashed boxes are the unmatched high-score detections D1 to D4. The matched tracks are treated as anchors with confidence 1 and run through NMS together with these detections: D4 overlaps T1 too much and is removed, and of D1 and D3, which overlap each other, only one is kept. On the right is the result: only two boxes become new tracks, T4 and T5.
Original Figure 4, Shim et al., from PDF page 5, scholarly quotation.

TAI takes the positions of the tracks already matched, treats them as boxes with confidence 1 that can't be removed, and runs NMS once more together with the leftover high-score boxes. Boxes that overlap too much with existing tracks, or with higher-scoring boxes, are removed. Only what remains becomes new tracks.

The threshold for this NMS can be adjusted per scene: set it lower, and only very certain new objects get created; set it higher, and in crowded scenes real newcomers are less likely to be missed.

The whole pipeline

Each frame first goes through YOLOX-X detection, an NSA Kalman filter predicts the track positions, and TPA does the matching. The leftover high-score boxes are then matched once more against new tracks "less than three frames old"; these new tracks are often noise such as reflections, and the authors don't let them compete with confirmed tracks for boxes. Finally, TAI decides which boxes may create new tracks.

Discussion

Test sets

DatasetTrackTrack HOTANext-best onlineBest offline
MOT1767.166.4 (ImprAsso)67.1 (CoNo-Link)
MOT2065.765.0 (AdapTrack)65.9 (CoNo-Link)
DanceTrack66.565.7 (Hybrid-SORT-ReID)63.8 (CoNo-Link)

MOT17 compiled from Table 1 of the paper, MOT20 from Table 2 of the paper, DanceTrack from Table 3 of the paper, all test sets.

On all three datasets, TrackTrack is first among online methods, and it's neck and neck with offline methods that get to see the whole video. The offline SUSHI scores 66.5 on MOT17 and 63.3 on DanceTrack, both below TrackTrack. On MOT17, TrackTrack's IDF1 is 83.1 and its AssA 68.2; the association-related metrics improve the most.

The authors particularly stress being "good on every dataset." Many methods are strong on MOT17 and fall off on DanceTrack, or the other way round; TrackTrack ranks near the top on all three.

Note that the detector is trained together with two extra pedestrian datasets, CrowdHuman and WiderPerson. SambaMOTR from the same period, which uses no extra data, has a HOTA of 67.2 on DanceTrack and isn't in this paper's comparison tables.

Ablations

SettingMOT17 HOTAMOT17 AssADanceTrack HOTADanceTrack AssA
Baseline67.369.661.947.7
+ TPA68.572.062.949.1
+ TAI67.469.662.648.7
Both69.172.763.349.7

Compiled from Table 4 of the paper, validation sets.

The baseline is the common approach: Hungarian algorithm, multi-stage matching, and a new track whenever the score clears the threshold. TPA contributes more than TAI, adding 2.4 AssA on MOT17.

TPA actually contains three changes, and the authors take them apart one by one:

  1. Mutual minimum vs. the Hungarian algorithm. Changing only the assignment method, with everything else the same, takes MOT17 HOTA from 68.1 to 69.1 and DanceTrack from 61.6 to 63.3.
  2. One pass vs. stages. MOT17 HOTA goes from 68.3 to 69.1; DanceTrack gains only 0.1, and its AssA actually drops by 0.1. Staging makes little difference on DanceTrack.
  3. Using boxes removed by NMS or not. With them, MOT17 HOTA goes from 68.6 to 69.1 and DanceTrack from 62.1 to 63.3.

The authors also analyze the effect of occlusion. Defining occlusion severity as "the number of overlaps with IoU above 0.6" divided by "the number of overlaps with IoU above 0.1," across five MOT17 and MOT20 videos, the heavier the occlusion, the more TrackTrack improves over the baseline. Under occlusion, the most suitable box is the most likely to be a low-score or removed box, and also the most likely to be split up by the Hungarian algorithm, which is exactly the case TPA was designed for.

Speed

TPA has to scan the whole matrix every round, which sounds like it would be slower than the Hungarian algorithm. Measured, it's actually a little faster. Excluding post-processing, it goes from 155.4 FPS to 161.5 FPS on MOT17 and from 394.8 to 408.4 on DanceTrack. The authors mention that TPA is written in plain Python while the Hungarian algorithm uses a heavily optimized library, so there's still room to speed it up. The paper states that post-processing was excluded from the measurement; YOLOX-X itself can't run at these speeds, so these numbers reflect the association step on its own.

Conclusion

TrackTrack changes how tracks and detection boxes get paired up, which is the least conspicuous link in a tracker.

The Hungarian algorithm seeks the lowest total cost; TPA seeks to give every track its most suitable box. Most of the time the two agree, and the difference shows up under occlusion: of two assignments whose total costs differ by 0.01, one swaps two people's identities and the other doesn't. Add low-score boxes and boxes removed by NMS into the same comparison, and occluded people stand a better chance of being found again.

Neither strategy needs training, and both can directly replace the association step of an existing tracker. With them, TrackTrack lets an online tracker that works frame by frame keep pace, on three datasets, with offline methods that see the whole video.