Back to paper library
arXiv 2023ICIP 2023Object trackingPaper notes

Deep OC-SORT

Bringing Appearance Back

Original paper

Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification

Paper authors

Gerard Maggiolino · Adnan Ahmad · Jinkun Cao · Kris Kitani

Publication
ICIP 2023Conference
First submitted to arXiv
arXiv:2302.11813
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingdeepocsort2023,
  title = {{Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification}},
  author = {Gerard Maggiolino and Adnan Ahmad and Jinkun Cao and Kris Kitani},
  year = {2023},
  eprint = {2302.11813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.11813}
}
On this page

Around 2022, trackers that associate by motion came back into fashion.

Detectors kept getting stronger, and ByteTrack, using only a Kalman filter and IoU, beat a pile of methods that learn appearance features on MOT17. OC-SORT fixed the Kalman filter's error buildup after occlusion, and in scenes like DanceTrack, where appearance is hard to tell apart and motion is complex, it also beat all kinds of re-ID methods.

But appearance cues aren't useless, just hard to use well. BoT-SORT and StrongSORT brought appearance back too, but in very simple ways. A track's appearance feature is updated with a fixed-weight exponential moving average, and the appearance distance is then added to the motion cost with a fixed weight.

Gerard Maggiolino and Adnan Ahmad of Carnegie Mellon University, together with Jinkun Cao and Kris Kitani, the authors of OC-SORT, want to add appearance to OC-SORT in a way that uses appearance features less when they're unreliable and more when they're reliable.

Because it adds deep appearance features to OC-SORT, it's called Deep OC-SORT. The authors make a point of saying that this isn't a rerun of what DeepSORT did to SORT.

Defining the problem

Appearance features extracted from a box are still very noisy, even with a strong re-ID model: a person is half hidden, fast motion causes blur, someone nearby happens to be dressed very similarly.

Previous approaches hard-code two things:

  • A fixed update weight. The exponential moving average's weight α\alpha is a constant, so whatever the detection quality in this frame, the new feature is mixed into the track's appearance model in the same proportion. A half-occluded box pollutes the track's appearance all the same.
  • A fixed fusion weight. The appearance cost is added to the motion cost in a fixed proportion. A detection that looks especially like one track and unlike all the others gets the same share of appearance cost as a detection that looks about equally like several tracks.

The authors want both weights to adapt to the situation. In addition, OC-SORT relies heavily on detection positions, and when the camera moves, positions get scrambled along with it, so they also add camera motion compensation.

Solving the problem

Original Fig. 1. The Deep OC-SORT pipeline. Along one path, the image goes into the ReID model to extract appearance features, and updates to the appearance features are controlled by the dynamic appearance (DA) module; along the other, it passes through camera motion compensation (CMC) and then into OC-SORT's Kalman filter. The results of the two paths are merged into a cost matrix by the adaptive weighting (AW) module and then handed to linear assignment to solve.
Original Fig. 1, Maggiolino et al., from PDF page 2, arXiv license, scholarly quotation.

Deep OC-SORT keeps OC-SORT's whole pipeline, including observation-centric momentum (OCM), observation-centric recovery (OCR) and online smoothing (OOS), as well as the Hungarian algorithm at the end. It adds three new modules: camera motion compensation (CMC), dynamic appearance (DA) and adaptive weighting (AW).

Camera motion compensation

Camera motion compensation uses OpenCV's VidStab module to estimate the similarity transform between two adjacent frames: it first extracts keypoints, computes optical flow, and then uses RANSAC to solve for a scale-rotation matrix Mt=stRtM_t = s_t R_t and a translation TtT_t.

BoT-SORT also does camera motion compensation, but corrects only the Kalman filter's state. OC-SORT uses past observed boxes directly in several more places, and all of these have to be corrected too:

  1. OOS: When an object reappears, OC-SORT linearly interpolates a virtual trajectory from the last observation and re-updates the Kalman filter. The center of the last observed box is first corrected with c←Mtc+Ttc \leftarrow M_t c + T_t before the interpolation starts.
  2. OCM: OC-SORT computes the direction of motion from the most recent Δt=3\Delta t = 3 boxes. The top-left and bottom-right corners of these boxes are corrected every frame with p←Mtp+Ttp \leftarrow M_t p + T_t.
  3. OCR: The last box seen is used to recover a lost track, and its position is corrected every frame as well.

The Kalman filter's state is x=[xc,yc,a,s,x˙c,y˙c,a˙]\mathbf{x} = [x_c, y_c, a, s, \dot{x}_c, \dot{y}_c, \dot{a}]: center position, area, aspect ratio, and the velocities of the first three. The correction is

x[0:2]←Mt x[0:2]+Ttx[4:6]←Mt x[4:6]P[0:2,0:2]←Mt P[0:2,0:2] Mt⊤P[4:6,4:6]←Mt P[4:6,4:6] Mt⊤\begin{aligned} \mathbf{x}[0{:}2] &\leftarrow M_t\,\mathbf{x}[0{:}2] + T_t \\ \mathbf{x}[4{:}6] &\leftarrow M_t\,\mathbf{x}[4{:}6] \\ P[0{:}2,0{:}2] &\leftarrow M_t\,P[0{:}2,0{:}2]\,M_t^\top \\ P[4{:}6,4{:}6] &\leftarrow M_t\,P[4{:}6,4{:}6]\,M_t^\top \end{aligned}

Only the center position and center velocity, and their covariances, are corrected. In theory, the transform's scaling could also be applied to the area and the rotation approximated as a change in aspect ratio, but the bounding box after rotation doesn't transform linearly, and computing it accurately would require the object's mask. The authors found that OCM and OCR tolerate this kind of approximation reasonably well, while the Kalman filter is much more sensitive, so they correct only the center. The correction is done before the Kalman filter's predict step.

Dynamic appearance

The usual exponential moving average is

et=α et−1+(1−α) enewe_t = \alpha\, e_{t-1} + (1 - \alpha)\, e^{new}

enewe^{new} is the appearance feature of the detection matched in this frame. The authors replace the fixed α\alpha with an αt\alpha_t that varies with detection confidence:

αt=αf+(1−αf)(1−sdet−σ1−σ)\alpha_t = \alpha_f + (1 - \alpha_f)\left(1 - \frac{s_{det} - \sigma}{1 - \sigma}\right)

sdets_{det} is the detection confidence, σ\sigma is the confidence threshold used to filter out noisy detections, and αf=0.95\alpha_f = 0.95.

The idea is straightforward. Low detection confidence usually means the object is occluded or the image is blurry, and the appearance feature extracted at that moment can't be trusted either.

  • When sdet=σs_{det} = \sigma, right at the threshold, αt=1\alpha_t = 1 and the new feature is ignored completely.
  • When sdet=1s_{det} = 1, αt=αf\alpha_t = \alpha_f and the new feature is added in the largest proportion.
  • Confidence values in between are linearly interpolated.

σ\sigma was already a tracker parameter, and αf\alpha_f is just the original fixed α\alpha. Dynamic appearance introduces no new hyperparameters, and its computational cost is practically zero.

Adaptive weighting

The appearance cost matrix AcA_c is M×NM \times N, for MM tracks and NN detections, and each cell is the cosine similarity between a track and a detection. The usual approach adds it to the IoU matrix with a fixed weight awa_w: C=IoU+awAcC = \text{IoU} + a_w A_c.

The authors' observation is that whether an appearance match can be trusted depends on whether it's "discriminative." A genuine appearance match has a clear gap between the highest and second-highest scores. If a track looks about equally like several detections, appearance shouldn't get much of a say.

For the mm-th track, discriminativeness is defined as the highest score in its row, am(1)a^{(1)}_m, minus the second highest, am(2)a^{(2)}_m, capped at ϵ\epsilon:

zmtrack=min⁡(am(1)−am(2), ϵ)z^{track}_{m} = \min\big(a^{(1)}_m - a^{(2)}_m,\ \epsilon\big)

For the nn-th detection, zndetz^{det}_{n} is computed the same way from its column. The average of the two is the extra weight for that cell:

wb(m,n)=zmtrack+zndet2w_b(m, n) = \frac{z^{track}_{m} + z^{det}_{n}}{2}

The final cost matrix is

C[m,n]=IoU[m,n]+[aw+wb(m,n)] Ac[m,n]\begin{aligned} C[m, n] = {} & \text{IoU}[m, n] \\ & + \big[a_w + w_b(m, n)\big]\, A_c[m, n] \end{aligned}

The authors deliberately look only at the top two instead of using a distribution measure like KL divergence. How much the lower scores further down differ from one another has nothing to do with whether this particular match can be trusted.

Implementation

The detector is the YOLOX used by ByteTrack and other methods, and the re-ID model is SBS50 from fast-reid.

On MOT17 and MOT20, aw=0.75a_w = 0.75 and ϵ=0.5\epsilon = 0.5. On DanceTrack, aw=1.25a_w = 1.25 and ϵ=1.0\epsilon = 1.0; the authors explain that appearance helps more than IoU there.

Discussion

What each module contributes

SettingMOT17 HOTAMOT17 IDF1MOT20 HOTADT HOTADT AssA
OC-SORT68.1379.5258.3553.0735.93
+CMC69.6682.4458.3353.5736.68
+App.68.5980.1859.1058.0342.37
+App. +DA68.6580.4559.1658.3643.00
+App. +DA +CMC69.8082.5659.3558.4643.33
All (+AW)70.2082.7859.4558.5343.41

Compiled from Table 3 of the paper, validation sets of the three datasets; DT = DanceTrack; "App." means appearance features updated with a fixed-weight EMA. MOT20 IDF1, row by row: 74.77, 74.63, 75.71, 75.87, 76.11, 76.30.

Camera motion compensation on its own adds nearly 3 points of IDF1 on MOT17 and a small gain on DanceTrack; MOT20 doesn't improve, because all MOT20 videos are shot with fixed cameras.

Appearance features are the biggest step on DanceTrack. Just adding appearance with a fixed EMA makes HOTA jump from 53.07 to 58.03, and AssA gains more than 6 points.

Dynamic appearance and adaptive weighting both give small gains, but they're positive on all three datasets and every metric.

Does appearance help on DanceTrack or not?

This result is very different from what the DanceTrack paper found. When the DanceTrack authors fixed the YOLOX detections and compared only association methods, DeepSORT was worse than SORT, and adding appearance made things worse; the oracle experiment with ground-truth boxes was the same, with every metric dropping once appearance was added.

Jinkun Cao is an author of both papers. Built on OC-SORT, though, appearance turned into the component that helps most on DanceTrack.

The two papers' settings are very different, with different re-ID models and different motion models. The DeepSORT in the DanceTrack paper uses appearance as its main basis for matching, on top of SORT's motion model, which easily loses accuracy after occlusion; Deep OC-SORT uses appearance as an auxiliary cost alongside IoU, added to a motion tracker that's already very stable. The dancers look very alike, but they aren't completely identical, and used the right way, appearance still has information to squeeze out.

Test sets

DatasetMethodHOTAMOTAIDF1ID sw.AssA
MOT17OC-SORT63.278.077.51,95063.2
MOT17StrongSORT++ (offline)64.479.679.51,19464.4
MOT17Deep OC-SORT64.979.480.61,02365.9
MOT20OC-SORT62.175.575.991362.0
MOT20StrongSORT++ (offline)62.673.877.077064.0
MOT20Deep OC-SORT63.975.679.277965.7

Compiled from Table 1 of the paper, MOT17 and MOT20 test sets. ID sw. = ID switches.

On MOT17, Deep OC-SORT's HOTA of 64.9 was second on the leaderboard at the time; on MOT20, its 63.9 was first. Compared with OC-SORT, its ID switches on MOT17 are cut almost in half.

StrongSORT++ uses offline post-processing (AFLink, GSI), while Deep OC-SORT is a purely online method, yet its association accuracy, AssA, is still higher.

Original Table 2. Comparison on the DanceTrack test set, with columns HOTA, DetA, AssA, MOTA and IDF1. The top half is methods that each use their own detections, with MOTR highest at HOTA 54.2; the bottom half uses the same YOLOX detections: SORT 47.9, DeepSORT 45.6, ByteTrack 47.3, OC-SORT 55.1, StrongSORT++ 55.6, and Deep OC-SORT at HOTA 61.3, DetA 82.2, AssA 45.8, MOTA 92.3, IDF1 61.5.
Original Table 2, Maggiolino et al., from PDF page 3, arXiv license, scholarly quotation.

On DanceTrack, among methods using the same YOLOX detections, Deep OC-SORT's HOTA is 61.3, 6.2 points above OC-SORT, and AssA rises from 38.3 to 45.8. DeepSORT is worse than SORT, while Deep OC-SORT is better than OC-SORT; with these two comparisons side by side in the same table, the difference in how appearance gets added is plain to see.

The authors claim this is the best result among published methods. But MOTRv2, posted to arXiv three months earlier, had already reported a much higher score on DanceTrack; it just hadn't been formally published yet, and it isn't in this table.

Limitations

This is a 4-page short conference paper, so the depth of analysis is limited.

All three new modules are hand-designed rules, and the appearance weight awa_w and cap ϵ\epsilon have to be tuned separately for different datasets; the DanceTrack settings are very different from MOT17's. The paper doesn't report speed either. With an extra re-ID model that has to extract features for every detection, it's bound to be slower than the purely motion-based OC-SORT.

Conclusion

Deep OC-SORT brings appearance back into OC-SORT, but with restraint. When detection confidence is low, appearance isn't updated; when appearance isn't discriminative, its weight isn't boosted; and when the camera moves, every old box in use gets corrected along with it.

Each module is small, but together they give steady gains on all three datasets.

The DanceTrack paper told everyone that when the dancers dress alike, appearance is useless. A little over a year later, one of its authors set the appearance weight on DanceTrack higher than on MOT17.