Back to paper library
arXiv 2018ICME 2018Object trackingPaper notes

MOTDT

Plan A and Plan B

Original paper

Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification

Paper authors

Long Chen · Haizhou Ai · Zijie Zhuang · Chong Shang

Publication
ICME 2018Conference
First submitted to arXiv
arXiv:1809.04427
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingmotdt2018,
  title = {{Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification}},
  author = {Long Chen and Haizhou Ai and Zijie Zhuang and Chong Shang},
  year = {2018},
  eprint = {1809.04427},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.04427}
}
On this page

If you've read the code of FairMOT or JDE, there's a good chance you've seen the name MOTDT in the association part.

The FairMOT paper is direct about it, writing "We follow MOTDT (Chen et al., 2018a) and use a hierarchical online data association method." Of the three classic association methods ByteTrack compares against, two are SORT and DeepSORT, and the third is MOTDT.

The name turns up in other people's tables all the time, yet the paper itself is only six pages long. Long Chen, Haizhou Ai and colleagues at Tsinghua University published it in 2018, and it deals with a problem every tracker runs into: what to do when the detector can't be relied on.

Defining the problem

In crowded scenes, detectors make mistakes easily. A change in pose or being blocked by someone else, and you get missed detections, false positives or boxes in the wrong place.

Offline (batch) methods can look at the whole video and use future frames to correct these errors, but they aren't real time and can't be used in applications that need results on the spot. This paper does online tracking only, using just the current and past frames.

In traditional tracking-by-detection, the only candidate boxes during association are the detections. If the detector misses someone, there's no box at all in that frame to give to their track.

The authors noticed that detection and tracking can actually complement each other:

  • Track predictions can cover short-lived missed detections or inaccurate boxes. If the detector doesn't find the person in this frame, the Kalman filter's predicted box is still there.
  • High-confidence detections can keep tracks from drifting onto the background over the long run. A predicted box that never gets corrected by a detection will go astray sooner or later.

So why not treat both as candidates? But detection boxes and predicted boxes come from different sources. How do you score them with the same yardstick, and then pick the best out of a pile of overlapping boxes?

Earlier, Yan et al. had also tried treating the outputs of both the detector and the tracker as candidates, but with hand-crafted features such as color histograms and optical flow. This paper wants to do it with deep networks, and in real time.

Solving the problem

Original Fig. 1. An example of candidate selection. The top row shows four consecutive frames (619, 622, 625, 628), each with a detection box as a solid blue line and a track-predicted box as a dashed red line; the middle row shows the candidates selected by the unified score. The curves below show how the scores of the two kinds of candidates change over the frames, with the solid blue line for the detection box and the dashed red line for the track-predicted box. Between frames 622 and 625, the detection box's score drops to around 0.8 while the track-predicted box's score stays near 1.0, so the predicted box is selected; at frame 628 the detection score recovers, and the selection switches back to the detection box.
Original Fig. 1, Chen et al., from PDF page 1, arXiv license, scholarly quotation.

This figure is the whole idea of the paper. The solid blue lines are detection boxes, the dashed red lines are the track's predicted boxes, and the curves below are the scores of the two.

In the frames before and after, the detection box scores higher and gets selected. In between, from frame 622 to 625, the person is partly occluded, the detection box's score drops, and the predicted box becomes the more reliable one, so the selection switches to the predicted box.

The method has two steps: first select candidates, then associate.

1. One classifier scores every candidate

Detection boxes plus predicted boxes add up to a lot of boxes, and they overlap heavily. Cropping each box and sending it through a classification network would be too slow.

Original Fig. 2. The R-FCN architecture used for classification. The input image (Image) goes through an encoder (Encoder) that extracts features, then a decoder (Decoder) with upsampling, where the encoder's features are concatenated with the upsampled features; the output is a set of score maps (Score maps), one per color, and RoI pooling (RoI Pooling) then gives each candidate box its score.
Original Fig. 2, Chen et al., from PDF page 3, arXiv license, scholarly quotation.

The authors switch to the R-FCN (region-based fully convolutional network) approach: the network runs only once over the whole image and outputs a few score maps, which all the candidate boxes share.

The backbone uses the lightweight SqueezeNet as its encoder, followed by a decoder with upsampling that raises the resolution of the score maps. For a 1152×6401152 \times 640 input image, it takes only 8 milliseconds on a GTX 1080 Ti.

The simplest way to score would be to compute, at every point in the image, the probability that it's "person," and then average all the points inside the box. But that throws away spatial information. If a box covers only half a person, the points inside it are still all on the person, so the average still comes out high.

R-FCN solves this with position-sensitive score maps. The network outputs k2k^2 score maps, each candidate box is divided into a k×kk \times k grid, and each cell reads only the score map that "belongs to its own position." Finally, the responses of the k2k^2 cells are averaged and passed through a sigmoid:

p(y∣z,x)=σ ⁣(1wh∑i=1k2 ∑(x,y)∈binizi(x,y))p(y \mid z, \mathbf{x}) = \sigma\!\left(\frac{1}{wh}\sum_{i=1}^{k^2}\ \sum_{(x,y) \in \text{bin}_i} z_i(x, y)\right)

Picture it with k=3k = 3. The network outputs 9 score maps, which respond to "a person's top-left corner," "a person's top center" ... "a person's bottom-right corner." When a box frames a person exactly, the top-left cell reads the top-left map, the bottom-right cell reads the bottom-right map, and every cell reads a high score. If the box is shifted half a person to the right, its right half lands on the background, the three cells on the right all read low scores, and the average drops.

This network only classifies; it doesn't regress boxes. In the implementation, k=7k = 7. The training data consists of positive samples randomly drawn around ground-truth boxes and an equal number of background negatives, taken from MS COCO and two MOT16 training videos.

2. Track confidence: how long can a predicted box be trusted?

Predicted boxes are useful for a short while, but over time they become unreliable. The longer the Kalman filter goes without a detection to update it, the less accurate its prediction.

The authors define a confidence score for each track. A track may break off and be recovered several times, and every recovery reinitializes the Kalman filter, so only the last continuous tracklet is considered:

strk=max⁡(1−log⁡(1+α⋅Ltrk), 0)⋅1(Ldet≥2)\begin{aligned} s_{trk} = {} &\max\big(1 - \log(1 + \alpha \cdot L_{trk}),\ 0\big) \\ &\cdot \mathbb{1}(L_{det} \geq 2) \end{aligned}

LdetL_{det} is the number of detections this tracklet has been matched to, and LtrkL_{trk} is the number of consecutive predictions used since the last time it was matched to a detection.

The formula behaves intuitively:

  • Right after a detection is matched, Ltrk=0L_{trk} = 0 and the confidence is 1.
  • Each additional prediction adds one to LtrkL_{trk}, and the confidence falls logarithmically.
  • Once 1+αLtrk1 + \alpha L_{trk} exceeds ee, the confidence hits zero, and this track's predicted box can no longer be a candidate.
  • A track has to be matched to at least two detections before there are enough observations to build a motion model, and only then can its predicted box be a candidate.

3. A unified score

The scores of the two kinds of candidates are defined as:

s=p(y∣z,x)⋅(1(x∈Cdet)+strk 1(x∈Ctrk))\begin{aligned} s = p(y \mid z, \mathbf{x}) \cdot \big(&\mathbb{1}(\mathbf{x} \in C_{det}) \\ &+ s_{trk}\,\mathbb{1}(\mathbf{x} \in C_{trk})\big) \end{aligned}

A detection box's score is simply the classification probability; a predicted box's score is the classification probability multiplied by the track confidence. Even if a predicted box looks very much like a person, its score gets pushed down if it comes from a track that hasn't been confirmed by a detection for a long time.

Once every candidate has been scored, non-maximum suppression is applied once (IoU threshold 0.3), and boxes scoring below 0.4 are discarded.

4. Appearance features

During association, deciding whether a candidate box and a track are the same person relies on re-ID features. The network uses the part-aligned architecture proposed by Zhao et al., with a GoogLeNet convolutional backbone followed by KK fully connected branches, each corresponding to a different body part. It outputs a 512-dimensional feature, compared by Euclidean distance.

The training data combines three person re-identification datasets, Market-1501, CUHK01 and CUHK03, and the network is trained with a triplet loss:

Ltriplet=1N∑⟨Ii,Ij,Ik⟩max⁡(dij−dik+m, 0)L_{triplet} = \frac{1}{N}\sum_{\langle I_i, I_j, I_k\rangle} \max(d_{ij} - d_{ik} + m,\ 0)

⟨Ii,Ij⟩\langle I_i, I_j\rangle is the same person, and ⟨Ii,Ik⟩\langle I_i, I_k\rangle are different people. Triplets that are already well separated (dik−dij>md_{ik} - d_{ij} > m) have zero loss, so training concentrates on the hard examples.

5. Hierarchical association

This is the part that JDE and FairMOT later adopted. Association happens in two tiers:

  1. Tier one: detection candidates × tracks, by appearance. The distance between re-ID features is computed, with a threshold of 0.4.
  2. Tier two: remaining candidates × remaining tracks, by IoU. The threshold is 0.3.
  3. Remaining detection boxes start new tracks.

Why split it into tiers? Predicted boxes are pushed forward by the Kalman filter, and in crowded scenes, the region a predicted box covers may include other people. Extracting appearance features from such a box can easily record someone else's looks. So appearance matching is used only on detection boxes, and a track's appearance is updated only from the detection boxes it's matched to; predicted boxes take part only in the second tier's IoU matching.

This is faster too. Each frame, re-ID features only need to be extracted once, for the detection boxes; predicted boxes need none at all.

Discussion

The experiments are run on MOT16 with the official public detections. A validation set taken from 5 training videos is used for the ablations.

What each component contributes

Original Table 1. Component ablation on the MOT16 validation set, where C is the classification probability, T the track confidence and A the re-ID appearance features. The baseline (Kalman filter plus IoU) has a MOTA of 28.4, IDF1 of 32.8, 628 ID switches and 0.85 false positives per frame; adding C gives 33.0, 37.6, 445 and 0.77; adding C and T gives 33.7, 37.3, 475 and 0.63; adding only A gives 30.6, 42.4, 234 and 1.01; adding all three gives 35.7, 45.3, 184 and 0.58.
Original Table 1, Chen et al., from PDF page 5, arXiv license, scholarly quotation.

The baseline is the plainest setup, Kalman filter prediction plus IoU matching, in other words SORT's recipe.

  1. Using the classifier to pick candidates from detection boxes and predicted boxes (C) raises MOTA from 28.4 to 33.0 and cuts ID switches from 628 to 445. Treating predicted boxes as candidates really does recover a lot of missed detections.
  2. Adding track confidence (T) lifts MOTA to 33.7, and false positives per frame drop from 0.77 to 0.63. Unreliable predicted boxes get suppressed, but ID switches rise slightly to 475.
  3. Adding only appearance features (A) lifts IDF1 sharply from 32.8 to 42.4 and cuts ID switches to 234, but MOTA is only 30.6, and false positives per frame actually rise to 1.01.
  4. Adding all three gives a MOTA of 35.7, an IDF1 of 45.3 and 184 ID switches, the best on all four metrics.

The first two mainly improve the detection-related metrics, and the appearance features mainly improve the identity-related ones. Only with both sides combined does everything fall into place.

The appearance features themselves were compared too. The table below, compiled from Table 2 of the paper, swaps in different appearance representations within the same framework:

AppearanceDimsMOTAIDF1ID switches
None–33.737.3475
Color histogram75034.938.6250
HOG115234.638.5317
Color + HOG190234.739.3307
re-ID feature51235.745.3184

Hand-crafted features help as well, but their IDF1 tops out at around 39; the learned re-ID feature has the fewest dimensions, yet its IDF1 is 6 points higher.

Speed

The authors measure per-frame processing time on the most crowded sequence, MOT16-03 (over 50 people per frame on average). Cropping every candidate box and classifying each one with SqueezeNet plus two fully connected layers takes 206.2 milliseconds per frame; switching to R-FCN with shared score maps brings it down to 66.3 milliseconds.

The MOT16 test set

Original Table 3. Results on the MOT16 test set, with every method using the same public detections. Among offline methods, LMP has a MOTA of 48.8, an IDF1 of 51.3, 481 ID switches and 0.5 FPS; JMC has 46.3 and 46.3. Among online methods, AMIR has 47.2 and 46.3, 2,681 FP, 774 ID switches and 1.0 FPS; STAM has 46.0 and 50.0, 473 ID switches and 0.2 FPS; MOTDT has 47.6 and 50.9, an IDR of 40.3, MT of 15.2%, ML of 38.3%, 9,253 FP, 85,431 FN, 792 ID switches and 20.6 FPS.
Original Table 3, Chen et al., from PDF page 6, arXiv license, scholarly quotation.

With public detections, MOTDT's MOTA of 47.6 and IDF1 of 50.9 are both the highest among the online methods in the table, and its ML of 38.3% and FN of 85,431 are even the lowest in the whole table. It runs at 20.6 FPS, which the authors put at roughly 5 to 20 times faster than most existing methods; most methods in the table manage only 0.2 to 1 frame per second, so the gap is actually larger. Compared with the best offline method, LMP, its MOTA is only 1.2 points lower, while it runs 40 times as fast.

Two columns don't look so pretty, though. FP is 9,253, more than three times AMIR's 2,681; ID switches are 792, also higher than STAM's 473 and LMP's 481. Our guess is that treating predicted boxes as candidates recovers missed detections but also keeps some boxes that shouldn't exist.

What came after

In their conclusion, the authors write that in the future they want the classifier and the appearance features to share convolutional layers, for further efficiency gains.

The following year, Zhongdao Wang and colleagues at Tsinghua University published JDE, which outputs detection boxes and re-ID features at the same time from a single detection network. Its official code also associates by matching first on appearance plus motion, then on IoU for whatever is left. The year after that, FairMOT adopted the same two-stage association. The other half of MOTDT, "treat predicted boxes as candidates too," wasn't carried over into FairMOT.

In 2021, ByteTrack included MOTDT in its comparisons, running it with the same YOLOX detector on the MOT17 validation set. MOTDT's MOTA is 75.8 and BYTE's 76.6. ByteTrack's authors argue that using predicted boxes as track boxes, as MOTDT does, may make positions drift little by little; BYTE instead fishes people back out of low-score detection boxes, which gives more accurate positions. The two papers tackle the same problem (how to keep from losing people when the detector is unreliable), but reach for different material to patch the holes.

Conclusion

MOTDT treats both detection boxes and track-predicted boxes as candidates, gives them a unified score with a position-sensitive fully convolutional classifier, and uses track confidence to push down predictions that haven't been confirmed for too long. During association, appearance is used only on detection boxes, and the rest is left to IoU. This approach reached the highest MOTA and IDF1 among online methods on MOT16 at the time, at 20 frames per second.

If you want to add re-ID to your own tracker, hierarchical association is a very practical starting point. Appearance features come only from trustworthy detection boxes, and predicted boxes are only there to fill in. The rule is simple, yet it directly avoids the mistake of "remembering someone else's face as your own."

As for "candidate selection," which takes up half the title, later trackers have hardly ever brought it up again.