FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking
- Publication
- International Journal of Computer Vision 2021Journal
- First submitted to arXiv
- arXiv:2004.01888
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingfairmot2020,
title = {{FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking}},
author = {Yifu Zhang and Chunyu Wang and Xinggang Wang and Wenjun Zeng and Wenyu Liu},
year = {2020},
eprint = {2004.01888},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2004.01888}
}
DeepSORT taught trackers to recognize people, at the cost of sending every detection box through a re-ID network once more. Ten people in the frame means cropping ten images and running the features ten times; a hundred people, a hundred times. The detector and the re-ID network each extract features from the raw image on their own, sharing nothing.
Once the crowd grows, this system slows down.
The natural idea is to put both jobs into one network. Around 2019, Track R-CNN attached a re-ID head to Mask R-CNN, and JDE (Joint Detection and Embedding) attached one to YOLOv3, so that a single forward pass outputs both boxes and features. Methods like these are called one-shot trackers. They really were faster, and their detection was decent too.
But ID switches shot up.
Yifu Zhang, Chunyu Wang and colleagues at Huazhong University of Science and Technology and Microsoft Research Asia wanted to find out what was going wrong. Their answer is that the two tasks weren't treated fairly inside the same network: detection was always the lead, and re-ID just came along for the ride. The first arXiv version of this paper was called "A Simple Baseline for Multi-Object Tracking"; only later did it get its current name, with its emphasis on "fairness."
Defining the problem
The authors trace one-shot trackers' failure to learn re-ID well to three kinds of "unfairness."
Unfairness caused by anchors

Track R-CNN and JDE are both adapted from anchor-based detectors, and their re-ID features follow the anchors too. That causes three problems:
- Re-ID is treated as a secondary task. Track R-CNN first uses an RPN to generate proposals, then extracts re-ID features from them. If the proposals are inaccurate, the features are useless, so during training the network learns the proposals first and puts re-ID second.
- One anchor box covers several people. In panel (b), the red box covers two sprinters at once. When RoI-Align samples inside the box, many sampling points land on someone else or on the background, so the extracted feature is a mix of several people.
- Several anchor boxes cover the same person. In panel (c), three anchor boxes in different positions all have a large enough IoU with the same sprinter, so all three are required to output the same identity. These three anchor boxes actually see different image patches, yet have to learn the same feature; and when data augmentation shifts the image slightly, the same anchor box may be assigned to someone else.
On top of that, a detector's feature maps are usually downsampled 8, 16 or 32 times. That's acceptable for detection but too coarse for re-ID, and the anchor's position may well be misaligned with the object's center.
Unfairness caused by features
Detection and re-ID want very different features. Detection wants objects of the same class to have features as similar as possible, so that a person is recognized as "a person" no matter what they're wearing; re-ID has to tell apart different individuals of the same class, which needs more low-level appearance detail.
When the two tasks share the same features, they pull against each other.
Unfairness caused by dimensions
The re-ID field is used to features of 512 or 1024 dimensions, while the detection head outputs just the number of classes plus a few box coordinates. The authors argue that with such a gap in dimensions, the re-ID loss overwhelms detection.
More importantly, re-ID in multi-object tracking is different from re-ID in general. General re-ID has to compare a query image against tens of thousands of candidates and needs very high discriminative power; tracking only needs a small number of one-to-one matches between two adjacent frames. Tracking has no use for such high dimensions.
Solving the problem

FairMOT takes on each of the three unfairnesses in turn. It drops anchors in favor of CenterNet-style detection, which represents objects by their center points; it uses a backbone with multi-layer feature fusion so that each task can take what it needs; and it lowers the dimension of the re-ID features.
The two branches have exactly the same structure and both attach directly to the same feature map output by the backbone, with neither coming before the other.
The backbone
The backbone is DLA-34, made of ResNet-34 plus the multi-layer fusion structure of Deep Layer Aggregation. All the convolutions in the upsampling modules are replaced by deformable convolutions, so the receptive field can adapt to an object's size and pose.
The output feature map is 1/4 the size of the input. In both training and inference, input images are resized to , which makes the feature map .
Following one person through
Let's use one person to see how the two branches divide the work on the same feature map.
Suppose that in a input image, a person's ground-truth box is .
Step one: find the center. The box's center is . The feature map is downsampled by 4, so the center falls at , which rounds down to cell on the feature map.
Step two: the heatmap. The heatmap branch outputs a map. The training target places a peak of value 1 at , falling off outward as a Gaussian. With objects, the target heatmap is
The loss takes the form of CenterNet's focal loss. The peak position should be predicted close to 1 and every other position close to 0, but the closer a position is to the peak, the less it's penalized for getting it wrong.
Step three: offset and size. Rounding threw away , and in general rounding can cost up to 4 pixels. At cell , the offset branch has to predict to restore the center's exact position, and the size branch predicts the box's width and height, , at the same cell. Both use an L1 loss, and the size term has a weight of , following CenterNet's setting.
Step four: re-ID. The re-ID branch uses a convolutional layer with 128 kernels to output , a 128-dimensional feature at every cell. This person's feature is the vector of at , .
This is FairMOT's key design. A person corresponds to just one point on the feature map, and the heatmap, offset, size and re-ID are all read out from that one point. No anchors means no "one anchor, two people" or "three anchors, one person."
How re-ID is trained
Re-ID training is treated as a classification problem. Each identity in the training set is a class, classes in all. The feature taken at an object's center goes through a fully connected layer and a softmax, giving the probability that the object belongs to each identity, and the loss is computed with cross-entropy:
Training uses only the features at ground-truth center points. At inference time, object centers are determined by the heatmap, and the features taken there are used directly for matching; the classification layer isn't needed.
Balancing the two tasks
The detection loss is the heatmap loss plus the box loss, . The two tasks' losses aren't simply added together; instead they use the uncertainty weighting proposed by Kendall et al.:
and are learnable parameters. The larger and more uncertain a task's loss, the more its weight gets turned down, but the terms penalize the weights for shrinking indefinitely. The network decides for itself how much each side gets.
Training on single images
Tracking data needs identity annotations across frames, which are expensive. The authors also propose "single-image training": take pedestrian detection data such as CrowdHuman, which has boxes but no identities, treat every box as a separate identity, and apply augmentations such as rotation, scaling, translation and color changes to the whole image.
Re-ID trained this way learns that "this person is different from everyone else in the frame." The paper calls the method self-supervised, while the official code's README calls it weakly supervised; whatever the name, it uses box annotations and saves on identity annotations.
Inference and association
At inference time, a max pooling on the heatmap serves as non-maximum suppression, keeping peaks that score above a threshold; boxes are then computed from the offset and size branches, and features are taken from the re-ID branch.
Association follows MOTDT's two-stage approach:
- Stage one: mainly appearance. A Kalman filter predicts each track's position, and the Mahalanobis distance is computed; then the cosine distance between the re-ID features of tracks and detections is computed. The two are weighted into with , and pairs whose Mahalanobis distance is too large are ruled out. Matching uses the Hungarian algorithm with a threshold of 0.4.
- Stage two: IoU picks up what's left. The remaining unmatched detection boxes and tracks are matched once more by box IoU, with a threshold of 0.5.
A track's appearance feature is updated every frame to adapt to changes in appearance. Detection boxes still unmatched at the end become new tracks; unmatched tracks are kept for 30 frames in case they come back.
Discussion
The training data is the same as JDE's and is called MIX. Within it, ETH and CityPersons have boxes only and train only the detection branch; CalTech, MOT17, CUHK-SYSU and PRW have identity annotations and train both branches. The ablations are all run on the MOT17 validation set.
Re-ID quality is measured by TPR@FAR=0.1: features taken at ground-truth box positions are used for retrieval, and this is the true positive rate at a false accept rate of 0.1. The metric isn't affected by detection results and looks only at how good the features themselves are.
Where to take the features

All five methods are built on FairMOT; the only difference is where the re-ID features are taken from. ROI-Align is Track R-CNN's approach, POS-Anchor is JDE's, and Center is FairMOT's; Center-BI uses bilinear interpolation at the center to get a more precise position, and Two-Stage detects first and then does ROI-Align on the detection boxes.
MOTA sits around 69 for nearly all of them, since the detection branch is the same. The differences are in IDF1 and ID switches. Center's 299 ID switches are about 30% fewer than POS-Anchor's 434, and Center-BI has the highest TPR. The worst is Two-Stage, with an IDF1 of just 68.2 and a TPR that falls to 90.5. Making re-ID depend on detection results during training really does drag down feature quality.
Do detection and re-ID really fight?
The authors ran a very direct experiment. With ResNet-34 as the backbone, one model trains both branches together, and another trains only the detection branch, leaving the re-ID branch at its random initialization.
Training detection alone raises AP from 75.1 to 76.1; training both tasks together really does hold detection back. Interestingly, the detection-only model even has a slightly higher MOTA (63.7 vs. 63.6), but IDF1 falls from 67.2 to 60.4 and TPR from 90.9 to 36.7. MOTA mainly reflects detection, so it can't tell when the re-ID features have become all but useless.
Switching to DLA-34, with its multi-layer fusion, gives an AP of 81.2 and a TPR of 94.4, both better than ResNet-34. The authors read this as multi-layer fusion letting the two tasks take the features they each need from different layers, which eases the conflict.
The backbone comparison supports this too. ResNet-50 is larger than ResNet-34 and more accurate on ImageNet, yet MOTA only goes from 63.6 to 63.7, and ID switches actually rise from 435 to 501; ResNet-34 with FPN has fewer parameters than ResNet-50, yet its TPR rises from 90.9 to 94.2. If DLA-34's deformable convolutions are replaced with ordinary ones, MOTA drops from 69.1 to 65.0 and AP from 81.2 to 78.1. A method that takes features from a single point at the center depends heavily on a receptive field that can adapt to the object's size.
A lower dimension is better
Reducing the re-ID features from 512 to 64 dimensions:
- JDE: MOTA 59.9 → 60.3, IDF1 64.1 → 65.0, ID switches 536 → 474, TPR 76.8 → 82.0. Every number improves.
- FairMOT: MOTA 68.5 → 69.2, AP 80.9 → 81.3, ID switches 312 → 283; but IDF1 dips slightly from 73.7 to 73.3, and TPR from 94.6 to 94.3.
FairMOT's results are more nuanced. High-dimensional features are indeed a little more discriminative on their own, but they drag down detection, and once detection suffers, overall tracking suffers with it. The authors settle on 128 dimensions.
How to weight the losses
Fixed weights (found by grid search) give the highest MOTA and AP and the lowest TPR, so the model leans toward detection; MGDA-UB gives the highest TPR and the lowest MOTA and AP, leaning toward re-ID. GradNorm has the highest IDF1 and is the most balanced overall, but it takes longer to train. The authors end up choosing uncertainty weighting, which is slightly behind but much simpler.
The three parts of association
With IoU-only association, there are 648 ID switches; with re-ID features only, 435; with re-ID plus a Kalman filter, 342; with all three, 299. In crowded scenes or with fast camera motion, IoU alone keeps switching IDs; when objects are partly occluded, the re-ID features become unreliable, and that's when boxes and motion have to fill in.
Single-image training
A model trained only on single CrowdHuman images, without ever seeing an identity annotation, gets a MOTA of 64.1 and an IDF1 of 64.9 when used directly on the MOT17 validation set. Pretraining on CrowdHuman and then fine-tuning on MOT17 gives a MOTA of 71.1 and an IDF1 of 75.6, a clear step above the 67.5 and 69.9 from training on MOT17 alone, and even higher than the 69.1 and 72.8 from MIX plus MOT17.
MOTChallenge

This table follows the private detector protocol, where everyone uses their own detector. FPS counts both detection and association, which is harsher on two-step methods, and exactly what one-shot trackers want to prove.
FairMOT ranks first on all four datasets. On MOT17, its MOTA of 73.7 and IDF1 of 72.3 are about 6 and 8 points above CenterTrack, another one-shot tracker, at 25.9 FPS. On MOT16, compared with DeepSORT (DeepSORT_2 in the table), MOTA rises from 61.4 to 74.9, and speed from under 6.4 FPS to 25.9.
The ID switch column deserves a closer look, though. On MOT16, FairMOT's 1074 are more than DeepSORT_2's 781. FairMOT detects more, with higher MT and lower ML, so it tracks more trajectories to begin with, and the absolute number of ID switches goes up with them. When comparing how well IDs are maintained, IDF1 is the more suitable metric.
On speed, FairMOT is down to 13.2 FPS on MOT20. When the authors broke down the time spent on each part, they found that network inference for detection and re-ID is barely affected by the number of people, and the Kalman filter and IoU matching take only 1 or 2 milliseconds; what grows linearly with the number of people is the time spent updating each track's appearance feature during re-ID matching.
What came after
Later critiques from the community mostly concern crowded scenes. CSMOT in 2023, for example, points out that in crowded scenes FairMOT's detection isn't accurate enough and its re-ID features aren't discriminative enough, so tracking performance drops noticeably. Our guess is that this is tied to the design of taking features from a single point at the center. When people are packed together, the area around one person's center point is often someone else's body.
Conclusion
One-shot trackers had a surge of ID switches, and FairMOT found three reasons. Anchors make re-ID learn muddled features, shared features make the two tasks pull against each other, and high-dimensional features overwhelm detection. The corresponding fixes are to represent objects by their center points, use a backbone with multi-layer fusion, and cut re-ID down to 128 dimensions. The result ranked first on four MOTChallenge datasets, with the whole system running at about 26 frames per second.
When you pack two tasks into one network, remember to measure the re-ID metrics separately. In the detection-only model, the features had become almost useless, yet MOTA didn't budge.
Funnily enough, a year and a half later, when Yifu Zhang wrote ByteTrack, re-ID was dropped altogether on MOT17.