Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification
- Publication
- ICIP 2023Conference
- First submitted to arXiv
- arXiv:2302.11813
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingdeepocsort2023,
title = {{Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification}},
author = {Gerard Maggiolino and Adnan Ahmad and Jinkun Cao and Kris Kitani},
year = {2023},
eprint = {2302.11813},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2302.11813}
}
Around 2022, trackers that associate by motion came back into fashion.
Detectors kept getting stronger, and ByteTrack, using only a Kalman filter and IoU, beat a pile of methods that learn appearance features on MOT17. OC-SORT fixed the Kalman filter's error buildup after occlusion, and in scenes like DanceTrack, where appearance is hard to tell apart and motion is complex, it also beat all kinds of re-ID methods.
But appearance cues aren't useless, just hard to use well. BoT-SORT and StrongSORT brought appearance back too, but in very simple ways. A track's appearance feature is updated with a fixed-weight exponential moving average, and the appearance distance is then added to the motion cost with a fixed weight.
Gerard Maggiolino and Adnan Ahmad of Carnegie Mellon University, together with Jinkun Cao and Kris Kitani, the authors of OC-SORT, want to add appearance to OC-SORT in a way that uses appearance features less when they're unreliable and more when they're reliable.
Because it adds deep appearance features to OC-SORT, it's called Deep OC-SORT. The authors make a point of saying that this isn't a rerun of what DeepSORT did to SORT.
Defining the problem
Appearance features extracted from a box are still very noisy, even with a strong re-ID model: a person is half hidden, fast motion causes blur, someone nearby happens to be dressed very similarly.
Previous approaches hard-code two things:
- A fixed update weight. The exponential moving average's weight is a constant, so whatever the detection quality in this frame, the new feature is mixed into the track's appearance model in the same proportion. A half-occluded box pollutes the track's appearance all the same.
- A fixed fusion weight. The appearance cost is added to the motion cost in a fixed proportion. A detection that looks especially like one track and unlike all the others gets the same share of appearance cost as a detection that looks about equally like several tracks.
The authors want both weights to adapt to the situation. In addition, OC-SORT relies heavily on detection positions, and when the camera moves, positions get scrambled along with it, so they also add camera motion compensation.
Solving the problem

Deep OC-SORT keeps OC-SORT's whole pipeline, including observation-centric momentum (OCM), observation-centric recovery (OCR) and online smoothing (OOS), as well as the Hungarian algorithm at the end. It adds three new modules: camera motion compensation (CMC), dynamic appearance (DA) and adaptive weighting (AW).
Camera motion compensation
Camera motion compensation uses OpenCV's VidStab module to estimate the similarity transform between two adjacent frames: it first extracts keypoints, computes optical flow, and then uses RANSAC to solve for a scale-rotation matrix and a translation .
BoT-SORT also does camera motion compensation, but corrects only the Kalman filter's state. OC-SORT uses past observed boxes directly in several more places, and all of these have to be corrected too:
- OOS: When an object reappears, OC-SORT linearly interpolates a virtual trajectory from the last observation and re-updates the Kalman filter. The center of the last observed box is first corrected with before the interpolation starts.
- OCM: OC-SORT computes the direction of motion from the most recent boxes. The top-left and bottom-right corners of these boxes are corrected every frame with .
- OCR: The last box seen is used to recover a lost track, and its position is corrected every frame as well.
The Kalman filter's state is : center position, area, aspect ratio, and the velocities of the first three. The correction is
Only the center position and center velocity, and their covariances, are corrected. In theory, the transform's scaling could also be applied to the area and the rotation approximated as a change in aspect ratio, but the bounding box after rotation doesn't transform linearly, and computing it accurately would require the object's mask. The authors found that OCM and OCR tolerate this kind of approximation reasonably well, while the Kalman filter is much more sensitive, so they correct only the center. The correction is done before the Kalman filter's predict step.
Dynamic appearance
The usual exponential moving average is
is the appearance feature of the detection matched in this frame. The authors replace the fixed with an that varies with detection confidence:
is the detection confidence, is the confidence threshold used to filter out noisy detections, and .
The idea is straightforward. Low detection confidence usually means the object is occluded or the image is blurry, and the appearance feature extracted at that moment can't be trusted either.
- When , right at the threshold, and the new feature is ignored completely.
- When , and the new feature is added in the largest proportion.
- Confidence values in between are linearly interpolated.
was already a tracker parameter, and is just the original fixed . Dynamic appearance introduces no new hyperparameters, and its computational cost is practically zero.
Adaptive weighting
The appearance cost matrix is , for tracks and detections, and each cell is the cosine similarity between a track and a detection. The usual approach adds it to the IoU matrix with a fixed weight : .
The authors' observation is that whether an appearance match can be trusted depends on whether it's "discriminative." A genuine appearance match has a clear gap between the highest and second-highest scores. If a track looks about equally like several detections, appearance shouldn't get much of a say.
For the -th track, discriminativeness is defined as the highest score in its row, , minus the second highest, , capped at :
For the -th detection, is computed the same way from its column. The average of the two is the extra weight for that cell:
The final cost matrix is
The authors deliberately look only at the top two instead of using a distribution measure like KL divergence. How much the lower scores further down differ from one another has nothing to do with whether this particular match can be trusted.
Implementation
The detector is the YOLOX used by ByteTrack and other methods, and the re-ID model is SBS50 from fast-reid.
On MOT17 and MOT20, and . On DanceTrack, and ; the authors explain that appearance helps more than IoU there.
Discussion
What each module contributes
| Setting | MOT17 HOTA | MOT17 IDF1 | MOT20 HOTA | DT HOTA | DT AssA |
|---|---|---|---|---|---|
| OC-SORT | 68.13 | 79.52 | 58.35 | 53.07 | 35.93 |
| +CMC | 69.66 | 82.44 | 58.33 | 53.57 | 36.68 |
| +App. | 68.59 | 80.18 | 59.10 | 58.03 | 42.37 |
| +App. +DA | 68.65 | 80.45 | 59.16 | 58.36 | 43.00 |
| +App. +DA +CMC | 69.80 | 82.56 | 59.35 | 58.46 | 43.33 |
| All (+AW) | 70.20 | 82.78 | 59.45 | 58.53 | 43.41 |
Compiled from Table 3 of the paper, validation sets of the three datasets; DT = DanceTrack; "App." means appearance features updated with a fixed-weight EMA. MOT20 IDF1, row by row: 74.77, 74.63, 75.71, 75.87, 76.11, 76.30.
Camera motion compensation on its own adds nearly 3 points of IDF1 on MOT17 and a small gain on DanceTrack; MOT20 doesn't improve, because all MOT20 videos are shot with fixed cameras.
Appearance features are the biggest step on DanceTrack. Just adding appearance with a fixed EMA makes HOTA jump from 53.07 to 58.03, and AssA gains more than 6 points.
Dynamic appearance and adaptive weighting both give small gains, but they're positive on all three datasets and every metric.
Does appearance help on DanceTrack or not?
This result is very different from what the DanceTrack paper found. When the DanceTrack authors fixed the YOLOX detections and compared only association methods, DeepSORT was worse than SORT, and adding appearance made things worse; the oracle experiment with ground-truth boxes was the same, with every metric dropping once appearance was added.
Jinkun Cao is an author of both papers. Built on OC-SORT, though, appearance turned into the component that helps most on DanceTrack.
The two papers' settings are very different, with different re-ID models and different motion models. The DeepSORT in the DanceTrack paper uses appearance as its main basis for matching, on top of SORT's motion model, which easily loses accuracy after occlusion; Deep OC-SORT uses appearance as an auxiliary cost alongside IoU, added to a motion tracker that's already very stable. The dancers look very alike, but they aren't completely identical, and used the right way, appearance still has information to squeeze out.
Test sets
| Dataset | Method | HOTA | MOTA | IDF1 | ID sw. | AssA |
|---|---|---|---|---|---|---|
| MOT17 | OC-SORT | 63.2 | 78.0 | 77.5 | 1,950 | 63.2 |
| MOT17 | StrongSORT++ (offline) | 64.4 | 79.6 | 79.5 | 1,194 | 64.4 |
| MOT17 | Deep OC-SORT | 64.9 | 79.4 | 80.6 | 1,023 | 65.9 |
| MOT20 | OC-SORT | 62.1 | 75.5 | 75.9 | 913 | 62.0 |
| MOT20 | StrongSORT++ (offline) | 62.6 | 73.8 | 77.0 | 770 | 64.0 |
| MOT20 | Deep OC-SORT | 63.9 | 75.6 | 79.2 | 779 | 65.7 |
Compiled from Table 1 of the paper, MOT17 and MOT20 test sets. ID sw. = ID switches.
On MOT17, Deep OC-SORT's HOTA of 64.9 was second on the leaderboard at the time; on MOT20, its 63.9 was first. Compared with OC-SORT, its ID switches on MOT17 are cut almost in half.
StrongSORT++ uses offline post-processing (AFLink, GSI), while Deep OC-SORT is a purely online method, yet its association accuracy, AssA, is still higher.

On DanceTrack, among methods using the same YOLOX detections, Deep OC-SORT's HOTA is 61.3, 6.2 points above OC-SORT, and AssA rises from 38.3 to 45.8. DeepSORT is worse than SORT, while Deep OC-SORT is better than OC-SORT; with these two comparisons side by side in the same table, the difference in how appearance gets added is plain to see.
The authors claim this is the best result among published methods. But MOTRv2, posted to arXiv three months earlier, had already reported a much higher score on DanceTrack; it just hadn't been formally published yet, and it isn't in this table.
Limitations
This is a 4-page short conference paper, so the depth of analysis is limited.
All three new modules are hand-designed rules, and the appearance weight and cap have to be tuned separately for different datasets; the DanceTrack settings are very different from MOT17's. The paper doesn't report speed either. With an extra re-ID model that has to extract features for every detection, it's bound to be slower than the purely motion-based OC-SORT.
Conclusion
Deep OC-SORT brings appearance back into OC-SORT, but with restraint. When detection confidence is low, appearance isn't updated; when appearance isn't discriminative, its weight isn't boosted; and when the camera moves, every old box in use gets corrected along with it.
Each module is small, but together they give steady gains on all three datasets.
The DanceTrack paper told everyone that when the dancers dress alike, appearance is useless. A little over a year later, one of its authors set the appearance weight on DanceTrack higher than on MOT17.