Tracking Objects as Points
- Publication
- ECCV 2020Conference
- First submitted to arXiv
- arXiv:2004.01177
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingcentertrack2020,
title = {{Tracking Objects as Points}},
author = {Xingyi Zhou and Vladlen Koltun and Philipp Krähenbühl},
year = {2020},
eprint = {2004.01177},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2004.01177}
}
In early April 2020, two tracking papers built on CenterNet appeared on arXiv two days apart.
One was FairMOT, which adds a re-ID branch to CenterNet so that the detector also outputs appearance features, then hands them to a DeepSORT-style pipeline for matching. The other is this paper, CenterTrack. Its first author, Xingyi Zhou, is the author of CenterNet, and the co-authors are Zhou's advisor at the University of Texas at Austin, Philipp Krähenbühl, and Vladlen Koltun of Intel Labs. They went in a different direction: instead of learning appearance, they let the detector look at two frames and, while it's at it, tell you where each object came from.
The same detector, modified in two completely different ways.
Defining the problem
In the early days of computer vision, tracking meant tracking interest points in an image, such as corners, and seeing where they moved in the next frame. This approach is simple and fast, but it fails wherever there are no distinct corners or intensity peaks.
Once deep detectors got strong, tracking-by-detection became the mainstream: detect in every frame first, then link the boxes up. The authors see two problems with this pipeline. First, the best trackers often rely on slow, complex association strategies, and those that use appearance features have to run yet another expensive feature network. Second, detection and tracking are separate, so when the detector looks for objects in frame , it knows nothing about what happened in frame .
Tracktor had already brought detection and tracking together, but it hands the previous frame's boxes to Faster R-CNN as proposals to regress, which assumes an object's boxes in two adjacent frames still overlap enough. In low-frame-rate videos, that premise doesn't hold, and Tracktor has to add a separate motion model.
If objects are represented as points, things get much simpler. CenterTrack starts from the idea of tracking an object's center point through time, going back to the early "track the points" way of thinking, except that the points are found by a detector.
The authors also draw their boundary early. CenterTrack only does local tracking, associating objects only between adjacent frames; an object that leaves the frame, or is occluded and then reappears, gets a new ID. It trades the ability to reconnect over long ranges for simplicity, speed and local accuracy.
Solving the problem
A quick refresher on CenterNet
CenterNet is a detector that represents objects as points. Given an image, it outputs a heatmap downsampled by a factor of 4, with one channel per class; each local maximum on the heatmap (the strongest point in its neighborhood) is the center of an object, and the height of the peak is the confidence score. Two other output branches read out, at the same position, the box's width and height and a local offset that makes up for the rounding error.
During training, each object's center is drawn as a Gaussian peak whose width scales with the object's size; the loss is a focal loss with and . Box size and local offset are supervised with an L1 loss at the center points only.
What CenterTrack looks at in addition

CenterTrack's network is almost identical to CenterNet's, with just four more input channels and two more output channels.
Extra inputs: the previous frame's image , three channels; and the centers of the objects already tracked in the previous frame, drawn as a single-channel, class-agnostic heatmap , one channel. To keep false positives from being passed along, only objects with a confidence score above a threshold are drawn into this heatmap.
Seeing the previous frame's image lets the network estimate how the scene has changed; seeing where the previous frame's objects were gives it a better chance of finding a person again, even if they're half occluded in this frame.
Extra output: a two-channel displacement map . At each detected center , the two-dimensional vector read out there, , is the object's displacement from the previous frame to this one:
It's trained the same way as box size, regressing the true displacement with an L1 loss at the ground-truth center points.
Following one car for one step
Once the displacement is predicted, association has hardly anything left to do. Let's walk through an example.
Suppose the previous frame has two tracks: A's center is at on the feature map, and B's at . In this frame, the network detects a center with a confidence score of 0.9, and the displacement read at this position is .
Working the displacement backwards, this object should have been at
in the previous frame. That's about 0.5 away from A and about 29.6 from B. As long as A hasn't already been claimed by someone else and the distance is within the radius , this detection is attached to A and inherits A's ID. is the geometric mean of the predicted box's width and height, , so larger objects are allowed larger errors.
All detections go through this greedy matching one after another, in descending order of confidence score. If there's no previous-frame object within the radius, a new track is started.
No Kalman filter, no appearance distance, and no Hungarian algorithm.
Compared with Tracktor, the difference is that the previous frame's objects enter the network as an input, in the form of a heatmap, rather than as proposals. The network reasons about the correspondence between objects in the two frames across its whole receptive field, so it can link them up even when the boxes in the two frames don't overlap at all.
The key to training: don't let it copy the answer
It sounds simple, but the authors say the details matter.
Objects in two adjacent frames are nearly identical. If the previous frame's heatmap were drawn straight from the ground truth during training, the network would quickly discover that copying the input heatmap to the output already gives a very low loss. What it would learn isn't tracking but copying.
At inference time, though, comes from the network's own predictions on the previous frame, complete with missed objects, misplaced objects and false positives. During training, the authors deliberately simulate these three kinds of errors:
- Position jitter. Gaussian noise is added to each previous-frame center, scaled to times the box's width and height.
- Fake peaks. With probability , an extra noise peak is drawn near a real object.
- Missing peaks. With probability , some objects are removed at random.
On MOT17, and , which means that on average 40% of the objects in the previous frame's heatmap are deliberately removed. On top of that, the "previous frame" isn't necessarily the real preceding frame; it's picked at random from within two frames before or after (), so that the network doesn't overfit to a particular frame rate.
This data augmentation is strong enough that CenterTrack can be trained without any video at all. Take a static image, randomly scale and translate it to serve as the "previous frame," simulate displacements from the original detection annotations, and you can train a tracker.
Other details
- 3D tracking. Switching to CenterNet's monocular 3D detection version and adding output heads for depth, rotation and 3D size gives monocular 3D tracking.
- Track rebirth. Unmatched tracks are kept for frames, during which they can still be rematched and get their original ID back, but they aren't drawn into the heatmap or output. Only the MOT17 test set uses this, with ; every other experiment uses .
Discussion
The authors evaluate on three datasets:
- MOT17: pedestrian tracking in crowded scenes, at 25 to 30 FPS. There's no official validation set, so the ablations split each training video into a first half and a second half.
- KITTI: a vehicle-mounted camera at 10 FPS, with large displacements between frames; only cars are evaluated.
- nuScenes: monocular 3D tracking in autonomous driving scenes, where the keyframes used for evaluation are only at 2 FPS and displacements between frames are very large. The main metric is AMOTA, a weighted average of MOTA across different confidence thresholds.
Main results
The official MOT17 leaderboard is open only for public detection. CenterTrack does its own detection by default, so the authors made a separate public version that follows Tracktor's rules: it keeps boxes close to boxes already tracked in the previous frame, and new tracks can only start near public detection boxes.
In this setting, MOTA is 61.5, 5 points above the best published method at the time, Tracktor v2 (56.5), and FN drops from 235449 to 200672. The authors give two reasons: the previous-frame heatmap makes already-tracked objects less likely to be missed, and the learned displacement makes association more stable. IDF1 is 59.6, lower than the 62.3 of the offline method LSST17 but higher than the other online methods.
The private version has a MOTA of 67.8 and an IDF1 of 64.7, at 57 milliseconds per frame including detection.
On KITTI, MOTA is 89.44, more than 4 points above the runner-up on the leaderboard, MASS (85.04). On the nuScenes test set, [email protected] is 27.8, against 6.9 for the official monocular baseline.

The same network, trained on a different dataset, handles pedestrians, vehicles and monocular 3D objects.
How much each design contributes

This table compares four settings:
- detection only: run CenterNet on each frame separately and associate by center-point distance.
- w/o offset: with the previous-frame heatmap, but with displacement set to zero, so it simply looks for the nearest object in the previous frame.
- w/o heatmap: with displacement prediction, but no previous-frame heatmap.
- Ours: both.
Start with the heatmap's effect. Comparing detection only with w/o offset, or w/o heatmap with Ours, adding the heatmap gains about 2 points of MOTA on MOT17 and about 3 on KITTI. FN goes down and FP goes up. With a hint from the previous frame, the network tends to find more objects near old peaks; sometimes that means recovering an occluded person, and sometimes it means being misled. Overall, it's still a good deal.
Now the displacement's effect. On the higher-frame-rate MOT17 and KITTI, MOTA is about the same with or without displacement, but ID switches drop noticeably (from 1.3% to 1.0% on MOT17, and from 1.6% to 0.1% on KITTI). On nuScenes, at 2 FPS, the gap opens up: w/o offset, with no displacement, gets only 17.8, while the full model gets 28.3. The lower the frame rate, the further objects move between two frames, and the less reliable "just find the nearest one" becomes.
Noise, static images, matching algorithms and rebirth

- Without noise, the model breaks. If heatmap errors aren't simulated during training, MOTA drops from 66.1 to 34.4, and FN shoots up from 28.4% to 57.4%. One reason the authors point to is that the input heatmap for the first frame is empty, and this model has only learned to find people by following the heatmap, so it struggles to discover new objects that aren't marked on it.
- Static images are enough. Trained on static images only, MOTA is the same 66.1, and IDF1 is even slightly higher. The authors also mention, though, that on the low-frame-rate nuScenes, training on static images works less well.
- The Hungarian algorithm doesn't help. Switching to globally optimal matching leaves MOTA unchanged, and IDF1 actually drops from 64.2 to 61.0. So the authors keep the simple greedy matching.
- Rebirth greatly improves IDs. Adding track rebirth with leaves MOTA almost unchanged, but IDF1 rises from 64.2 to 69.4 and ID switches drop from 1.0% to 0.4%. In scenes like MOT17, where occlusion is frequent, the identities that purely local tracking loses actually matter quite a lot.
What the learned displacement does beyond a motion model

The authors replace the learned displacement with three motion models: no motion (zero displacement), SORT's Kalman filter, and FlowNet2 optical flow (reading the flow vector at each predicted center). The detector is exactly the same.
On MOT17, the four barely differ; even using no motion model at all does about as well. At 30 FPS, a pedestrian moves only a tiny bit per frame, so any guess gets close enough.
On KITTI and nuScenes, the Kalman filter's hand-crafted motion rules clearly fall short; on nuScenes it gets only 18.3, almost the same as no motion. Optical flow is much better (26.6) but still loses to CenterTrack's 28.3, and FlowNet2 takes about 150 milliseconds per image pair.
The authors' explanation is that CenterTrack's displacement does more than estimate motion. The network can see where the previous frame's objects were, and training keeps pushing it to get things right on a noisy heatmap, so what it learns is to "snap" the displacement onto an object that really exists in the previous frame. The displacement takes care of motion estimation and association at the same time.
What came after
The FairMOT paper, commenting on CenterTrack, says it doesn't extract appearance features and only associates adjacent frames, while FairMOT itself can use appearance for long-range association and handle occlusion. Not reinitializing lost long-range tracks was a boundary CenterTrack had drawn for itself in its own introduction, and the rebirth experiment above shows what it costs: keeping lost tracks for 32 frames raises IDF1 from 64.2 to 69.4. In their discussion of the MOT17 results, the authors also mention that combining local trackers like this one with long-range re-ID methods is a direction worth pursuing.
Later, ByteTrack used CenterTrack as one of the baselines whose association could be swapped out, and on low-frame-rate data such as KITTI, methods extending CenterTrack kept appearing. Linking detections across two frames by displacement became a standard component of the "detection and tracking in one" line of work.
Conclusion
Represent objects as points, let the detector look at one more image from the previous frame plus a heatmap of the previous frame's objects, and have it output one more displacement map. CenterTrack's association comes down to "work backwards, find the nearest," yet it beat the published methods of the time on MOT17, KITTI and nuScenes.
The data augmentation that deliberately breaks the heatmap during training can go straight into your own projects. Without it, MOTA falls from 66 to 34; with it, even static images can be used to train a tracker.
A year and a half later, ByteTrack took nothing but CenterTrack's detection boxes, swapped in its own association and kept lost tracks for 30 extra frames, and IDF1 went up by nearly 10 points.