Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking
- Publication
- CVPR 2023Conference
- First submitted to arXiv
- arXiv:2203.14360
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingocsort2022,
title = {{Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking}},
author = {Jinkun Cao and Jiangmiao Pang and Xinshuo Weng and Rawal Khirodkar and Kris Kitani},
year = {2022},
eprint = {2203.14360},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2203.14360}
}
ByteTrack showed that with a strong enough detector, a Kalman filter plus IoU matching can take first place on MOT17.
Switch to DanceTrack, though, and it's a different story. The dataset films group dances, with dancers in nearly identical clothes spinning, crossing, jumping and swapping places. Detection isn't hard at all; what's hard is that nobody moves in a straight line. ByteTrack's HOTA here is under 48, well behind the end-to-end MOTR.
So have trackers that rely on motion models reached the end of the road?
Jinkun Cao, Kris Kitani and colleagues at Carnegie Mellon University don't think so. In their view, the Kalman filter itself is fine; the trouble is how SORT uses it. Handle the error that builds up over "the frames where the target can't be seen" properly, and the most basic Kalman filter can still reach the best results of its time.
This paper takes SORT apart and examines it from the ground up. The tracker it ends up with doesn't need to train any network, and given the detection boxes, it runs at over 700 frames per second on a single CPU.
Defining the problem
First, a quick review of how SORT uses the Kalman filter.
Each target's state is : the box center, area and aspect ratio, plus the velocities of the first three. In every frame, the filter first predicts, pushing the state forward with a constant-velocity model:
If a detection box is matched to it in this frame, the filter then updates with that box, correcting the prediction.
The problem comes when there's no detection box, for instance when the target is blocked. SORT simply takes the prediction as the updated result:
The authors call this step a "dummy update." The idea behind it is that when there's no observation, you trust your own estimate. The track is always extended by the estimate, and detection boxes are just auxiliary information for correcting it. The authors call this design estimation-centric.
Rethinking SORT's limitations
Starting from this design, the authors point out three interlocking problems.
1. Too sensitive to state noise.
Suppose the estimated center position carries noise with standard deviation , independent from frame to frame. Velocity is the difference between two positions divided by the time interval:
The smaller is, the noisier the velocity.
And multi-object tracking videos happen to have high frame rates. The authors measured the MOT17 training set: between two adjacent frames, targets move on average only 1.93 pixels horizontally and 0.65 pixels vertically. Let's do the math. If position noise is only 1 pixel, the velocity computed from two adjacent frames has a noise standard deviation of pixels per frame, about as large as the true velocity.
This is a real paradox. The higher the frame rate, the more reasonable the linear constant-velocity assumption; but the displacement between two frames also gets smaller, down to the same order of magnitude as the noise, so the estimated velocity actually gets less reliable.
2. Error grows over time.
When there are detection boxes, the next update can pull the position back even if the velocity estimate is off. But once the target is blocked, the Kalman filter can only keep pushing forward with dummy updates. After blocked frames:
The standard deviation of the position error is proportional to . With 1 pixel of position noise and 10 blocked frames, the position's standard deviation is pixels. The analysis in the appendix also points out that a real Kalman filter has process noise on the velocity terms too, so error accumulates even faster than in this simplified version. In a crowded scene, drifting by a dozen-odd pixels may well be enough to land the box on the person next door.
3. Estimation-centric.
The root of the first two problems is that SORT puts estimation ahead of observation. But a detector's noise is roughly independent from frame to frame and doesn't accumulate over time, and modern detectors are very accurate; the localization error in a single frame is usually smaller than that of the estimate the Kalman filter has been pushing forward all along.
In other words, the detector has told you plainly "the person is here," yet the tracker still follows its own estimate and extends the track somewhere else.
That's putting the cart before the horse.
Why a track gets lost again right after being recovered
The appendix has one more, finer observation that explains why these three problems are especially troublesome in practice.
In theory, once a target reappears and is matched to a detection box, the Kalman filter's update should correct the accumulated bias. But in practice, SORT sets the process noise on the velocity terms very low (only 0.01 in the official code, and just 0.0001 for the area's velocity), because velocity estimates are too sensitive to begin with and jump around if not held down. The price is that the filter corrects velocity very conservatively.
As a result, by the time the target has been blocked for a while, the direction of its velocity has long since drifted. The one update when it's matched again pulls the position back, but leaves the velocity's direction almost unchanged. The next frame predicts along the wrong direction, and the target is lost again.
So instead of trusting the estimate more, would it be a good idea to make tracking observation-centric?
Solving the problem

OC-SORT (Observation-Centric SORT) keeps SORT's whole skeleton: the Kalman filter, IoU and the Hungarian algorithm are all still there. It changes three places to be observation-centric.
Let's follow ballet dancer 1 in the figure:
- frame t: Dancer 1 is tracked normally. Matching adds an "is the direction of motion consistent?" cost on top of IoU. This is OCM.
- frame t+1: Dancer 1 is blocked by other dancers, no detection box is matched to them, and the track becomes temporarily lost (blue box).
- frame t+2: Dancer 1 reappears. The Kalman filter's prediction has already drifted and doesn't line up with the new detection box; but OC-SORT takes dancer 1's last observation, from frame t, and tries once more against the unmatched detection boxes, finding them again. This is OCR.
- After recovery: Dancer 1 comes back from the lost state, which triggers ORU, rewinding the Kalman filter's parameters over the period from frame t to t+2 and recomputing them.
Observation-Centric Re-Update (ORU)
ORU deals specifically with the problem of "lost again right after being matched again."
When a track is matched to a detection box again after being lost for a while, ORU goes back and examines the lost period. Let the last observation seen be and the observation it's matched to again be . Using these two real observations as endpoints, it generates a virtual trajectory in between:
where .
This is just linear interpolation under the constant-velocity assumption. Then the Kalman filter starts again from and reruns the "predict, update" loop along these virtual observations:
A numerical example shows how this differs from SORT (the numbers are made up for illustration). Suppose someone is last seen in frame 10 with their center at , is blocked for 5 frames, and reappears in frame 16 at . They sped up while they were blocked, but the velocity the Kalman filter holds is still the old 2 pixels per frame.
- SORT: Frames 11 to 15 are all dummy updates, and the prediction gets pushed to by frame 16. After being matched with 130, it does only one update; because the noise on the velocity term is set so low, the position is pulled back but the velocity stays close to 2. In the next frame, it guesses the wrong place again.
- ORU: With 100 and 130 as endpoints, it generates virtual observations for frames 11 to 15: 105, 110, 115, 120, 125. The Kalman filter walks through this stretch again, sees a movement of 5 pixels per frame at every step, and the velocity is pulled toward 5 step by step.
ORU doesn't make the prediction smarter. Before the filter sets off again, it fills in the middle using the "visible head and tail" and gets a few more updates in.

This is a real example from the paper. A member of the dragon dance team is blocked between the second and third moments and found again at the third. Without ORU, the estimated center (white stars) drifts off along the old direction, and by the fourth moment (gray star) it no longer matches the observation; the estimate after ORU's correction (yellow stars) stays on the person throughout.
Observation-Centric Momentum (OCM)
Besides "velocity doesn't change," the constant-velocity assumption also implies "direction doesn't change." A person walking left is unlikely to be suddenly matched to a box on the right in the next frame. SORT can't use this cue, because its velocity direction is computed from the estimate, which is too noisy.
OCM computes direction from observations instead, and adds directional consistency to the matching cost matrix:
is the negative IoU, is the direction difference, and . is the past observations of all existing tracks, and is this frame's new detections.

In the figure, the green line is an existing track and the red dots are this frame's new detections. For each pair of track and detection, two directions are computed:
- (blue line): the direction connecting two observations on the track frames apart, that is, the way this track has been heading.
- (yellow line): the direction from the track's past observation to this new detection, that is, where it would head next "if this detection were matched to it."
Direction is computed in radians as , and the difference between the two, , is the direction cost. The more consistent the directions, the lower the cost and the easier it is to match.
There's a trade-off here. Appendix A derives the probability distribution of the direction noise and concludes that the farther apart the two observations are, the less noisy the direction estimate, so should be somewhat large; but if they're too far apart, the linear-motion assumption no longer holds. In the implementation, the authors use .
Observation-Centric Recovery (OCR)
Last comes a small empirical trick. After normal association is finished, each track that's still unmatched takes its last observation and does one more round of IoU matching against the detection boxes that are still unmatched.
This step doesn't rely on the Kalman filter's prediction, so it's especially useful when the target stops, or is blocked for only a moment. The prediction has already run off, but the person is actually still near where they were.
Discussion
The authors evaluate on MOT17, MOT20, KITTI and DanceTrack, with results on the head-tracking dataset CroHD in the appendix.
To compare tracking algorithms alone, OC-SORT doesn't train its own detector. MOT17, MOT20 and DanceTrack use ByteTrack's public YOLOX weights directly, and KITTI uses PermaTrack's public detections. All datasets share one set of parameters: a detection score threshold of 0.6 (0.4 for MOT20) and an IoU threshold of 0.3.
The main metric is HOTA, with extra emphasis on AssA and IDF1, which measure association. The authors specifically point out that MOTA is so closely tied to detection quality that it's only suitable for comparison when everyone uses the same detections; that's also why, at the bottom of every table, the methods sharing detections are boxed off as a separate group.
DanceTrack results

DanceTrack is this paper's home turf, so let's look at it first.
The four methods in the light-blue block use the same detections. SORT's HOTA is 47.9 and ByteTrack's is 47.3, while DeepSORT, with appearance features added, is even lower at just 45.6. The dancers dress alike, so appearance doesn't help. OC-SORT gets 54.6, with AssA jumping in one go from around 31 to 40.2, and its IDF1 is also the highest of the four.
Now compare it with MOTR at the top of the table. MOTR is an end-to-end Transformer tracker with HOTA 54.2 and AssA 40.2; OC-SORT, without training any tracking network, has HOTA 54.6 and an AssA of 40.2 as well. The two use different detectors and OC-SORT's DetA is much higher, but their AssA, which measures association, is exactly the same. On the dataset that tests association hardest, a tracker built purely on a motion model draws level with an end-to-end learned method.
MOT17 results

Pedestrians in MOT17 mostly walk in straight lines, so SORT's assumption is fairly reasonable here to begin with. Against ByteTrack, which shares its detections, OC-SORT's HOTA is 63.2 vs. 63.1 and its IDF1 77.5 vs. 77.3, almost a tie; ID switches fall from 2196 to 1950.
But MOTA is 2.3 points lower (78.0 vs. 80.3). The reason is in the FP and FN columns. OC-SORT has far fewer false positives (1.51 vs. 2.55, in units of 10,000) but far more missed detections (10.8 vs. 8.37). ByteTrack recovers low-score boxes for matching, while OC-SORT follows the usual practice of throwing low-score boxes away and doesn't adjust the threshold separately. The missed boxes show up directly in MOTA.
MOT20 results

MOT20 is far more crowded, with far more occlusion, which is exactly the setting where accumulated error is worst. Here OC-SORT's advantage opens up. HOTA is 62.1 vs. 61.3, AssA 62.0 vs. 59.6, and ID switches fall from 1223 to 913; its HOTA was the highest on MOT20 when the paper was published. MOTA is again lower than ByteTrack's because of more missed detections.
On MOT17 and MOT20, both tables keep the baseline methods' linear-interpolation post-processing. To rule out the detector's influence, the authors also report results with the official public detections in the appendix, and OC-SORT leads in that setting too.
KITTI results

KITTI is footage from a car-mounted camera at only 10 FPS, and the results are half good, half bad.
For pedestrians, with the same detections, OC-SORT's HOTA is 52.95, more than 5 points above PermaTrack's 47.43, and it's 10 times faster too. The HP in the table is offline "head padding." Tracks in the SORT family only start producing output after being matched for several consecutive frames; these early boxes that weren't output are written back in after online tracking finishes, and with them filled in, pedestrian HOTA is 54.69.
Cars are another matter. OC-SORT's HOTA is 74.64, and only 76.54 even with HP, both below PermaTrack's 77.42. The authors' explanation is very frank. Cars move fast and the frame rate is low, so boxes in adjacent frames barely overlap and the IoU is close to zero, and IoU is exactly what OC-SORT's default association relies on. The problem lies outside the observation-centric design; every method that associates by IoU runs into it, and adding other association cues or appearance features can ease it.
Module contribution analysis

The three modules are added in turn:
- ORU helps on both datasets. HOTA on the MOT17 validation set rises from 64.9 to 66.3, and on DanceTrack from 47.8 to 48.5.
- OCM's effect depends on the scene. On DanceTrack, OCM makes HOTA jump from 48.5 to 52.1, the biggest step of the three modules; on MOT17, it only goes from 66.3 to 66.4. When pedestrians mostly walk in straight lines, directional consistency carries little information to begin with; it only comes into its own when dancers are spinning around.
- OCR is the finishing touch. MOT17's HOTA gains 0.1; DanceTrack's HOTA stays the same, while IDF1 rises from 50.6 to 51.6.
How ORU generates the virtual trajectory

Does the virtual trajectory have to be a straight line? The authors tried constant acceleration, linear regression and Gaussian process regression (GPR) with an RBF kernel.
The simplest option, constant velocity, turns out best on all six numbers, with constant acceleration second; linear regression and GPR are clearly worse, and GPR even loses 3.4 points of HOTA on MOT17. The authors suspect that because ORU runs online, with only a limited history of observations at hand, global fitting methods struggle to produce reliable trajectories; a local assumption that looks only at the two endpoints is actually more stable.
The appendix also compares offline interpolation post-processing, with a similar conclusion. On the MOT17 validation set, linear interpolation raises HOTA from 66.5 to 68.0, while GPR interpolation drops it to 65.2. Interestingly, StrongSORT, from the same period, also fills gaps with a Gaussian process in its GSI and does get an improvement; the two papers' settings differ, and GSI adjusts the kernel width according to track length. The same tool, used differently, can give very different results.
Choosing Δt for OCM

This table confirms the trade-off from earlier. As goes from 1 up to 2 and 3, performance improves on both datasets, because the direction estimate gets less noisy.
But stretch it to 6 and MOT17's HOTA falls back from 66.5 to 66.0. Over too long an interval, the target has long since stopped moving in a straight line. On DanceTrack, has a slightly higher HOTA (52.2), while and 6 are both 52.1, a very small difference. In the end the authors use on all datasets.
Speed
Given the detection boxes, OC-SORT runs at 793 frames per second on a single Intel i9-9980XE CPU. The three new modules are all just geometric computations with no neural network at all, so they add almost no cost.
Afterward
The authors mention in the appendix that OC-SORT is a pure motion model but can easily take appearance cues. Deep OC-SORT, published by the same lab the following year, adds adaptive re-ID to OC-SORT and improves further on MOT17, MOT20 and DanceTrack. Hybrid-SORT in 2023 keeps OC-SORT's observation-centric momentum and association, and adds weak cues such as the direction of box corners, detection confidence and height.
BoT-SORT, later the same year, also put OC-SORT in its comparison tables. It chose to add camera motion compensation to ByteTrack, dealing with a different kind of "prediction drift": the person isn't moving, the camera is.
Conclusion
OC-SORT doesn't replace the Kalman filter; it only changes how the filter behaves when the target can't be seen. When the target comes back, ORU uses the two real observations at the head and tail to fill in the middle and recalibrate the velocity; OCM uses directions computed from observations to help with matching; OCR uses the last observation to look once more. The result is that on DanceTrack, a motion-model tracker that needs no training draws level with the end-to-end MOTR.
For applications, the lesson of this paper can be boiled down to one sentence.
When the detector is already reliable, the core of tracking lies not in a more complex prediction model, but in better leveraging observations.
If the system you have is already SORT or ByteTrack, none of these three modules requires retraining anything. Just remember that once cars get fast and the frame rate gets low, IoU matching is still the first thing to give way.
For the frames where someone is blocked, rather than letting the filter guess blindly all the way through, wait for the person to come back, then rewind and recompute.