Back to paper library
arXiv 2021CVPR 2022Object trackingPaper notes

TrackFormer

One Query per Person

Original paper

TrackFormer: Multi-Object Tracking with Transformers

Paper authors

Tim Meinhardt · Alexander Kirillov · Laura Leal-Taixe · Christoph Feichtenhofer

Publication
CVPR 2022Conference
First submitted to arXiv
arXiv:2101.02702
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingtrackformer2021,
  title = {{TrackFormer: Multi-Object Tracking with Transformers}},
  author = {Tim Meinhardt and Alexander Kirillov and Laura Leal-Taixe and Christoph Feichtenhofer},
  year = {2021},
  eprint = {2101.02702},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.02702}
}
On this page

Tracktor showed in 2019 that a detector's box regression head can push the previous frame's boxes straight into the current frame, so tracking needs almost no separate training. CenterTrack swapped in center points and offsets and followed the same route. The authors call this kind of method "tracking-by-regression."

But to reach top results, methods like these all end up bolting something else on. Tracktor needs re-ID and a motion model; other methods hook up graph optimization. The reason is that each box is regressed independently and locally. A box doesn't know who it is, and it doesn't know what the other tracks around it are doing.

In 2020, DETR framed object detection as set prediction. A set of learnable object queries goes through a Transformer decoder and directly outputs a set of non-duplicate boxes, with no anchor boxes and no non-maximum suppression.

The two authors of Tracktor, Tim Meinhardt and Laura Leal-Taixé of the Technical University of Munich, teamed up with Alexander Kirillov and Christoph Feichtenhofer of Facebook AI Research to carry this idea over to tracking. If a query can represent an object, then letting it follow that object into the next frame would be tracking, wouldn't it?

Defining the problem

A video contains KK objects. Each object kk has a track Tk=(bt1k,bt2k,… )T_k = (b^k_{t_1}, b^k_{t_2}, \dots) recording its box in every frame from when it enters the scene to when it leaves, including the frames in which it is occluded.

The authors frame multi-object tracking as a frame-by-frame set prediction problem. In every frame, the model has to output a set of boxes with identities, covering both where the objects already being tracked are now and which objects are new.

Detection, keeping identities and starting new tracks all happen in the same decoding pass. There's no separate matching step and no graph optimization, and appearance and motion don't need to be modeled explicitly. The authors call this new paradigm "tracking-by-attention."

Solving the problem

Original Figure 1. An illustration of TrackFormer. Three consecutive frames pass through a CNN and a Transformer in turn. Each square is an output embedding corresponding to one person in the frame. The first frame has three people, red, green and blue; the second frame carries on these three identities and adds a newly appeared purple one; in the third frame the blue person has left, leaving only purple, red and green. Arrows send the previous frame's output embeddings into the next frame's Transformer.
Original Figure 1, Meinhardt et al., from PDF page 1, CC BY 4.0.

Architecture

TrackFormer's backbone is simply Deformable DETR:

  1. ResNet-50 extracts image features.
  2. A 6-layer Transformer encoder runs self-attention over the image features.
  3. A 6-layer Transformer decoder runs self-attention among the queries, then cross-attention to the encoded image features.
  4. An MLP maps each output embedding to a box and a class.

Everything that differs from DETR is in the decoder's queries.

Two kinds of queries

Original Figure 2. TrackFormer's frame-by-frame pipeline. At t = 0, the decoder turns a set of object queries (white) into output embeddings; embeddings that detect an object become colored track queries, and the rest are predicted as background (crossed out). At t − 1 and t, the decoder processes the object queries together with the track queries passed on from the previous frame, continuing or removing existing tracks, and detects new objects (purple) with the object queries.
Original Figure 2, Meinhardt et al., from PDF page 4, CC BY 4.0.

Object queries, as in DETR, are a fixed set of learnable embeddings responsible for discovering new objects in any frame. MOT17 has at most 52 people per frame, and the authors use 500 object queries; there need to be far more queries than objects in the scene for it to work well.

Track queries are the heart of this paper. When an object query detects an object in some frame (a classification score above 0.4), its output embedding from the decoder's last layer becomes a track query for the next frame directly, carrying an identity number with it.

In the next frame, the decoder processes Nobject+NtrackN_{object} + N_{track} queries at once. Through cross-attention, each track query looks for its object in the new image and outputs an updated box; the updated embedding is then passed on to the frame after that. Identities are passed along autoregressively like this, frame by frame, without any matching at any point.

The self-attention between the two kinds of queries matters too. Object queries can "see" the track queries, so they don't re-detect people who are already being tracked.

To make it easier for track queries to find their objects, the implementation has two more details. The reference point of a track query in deformable attention is set directly to the center of the previous frame's box, and the decoder's cross-attention looks at both the previous and current frames' feature maps, telling them apart with a temporal encoding.

Ending tracks and getting them back

A track query is removed when its classification score drops below 0.4, or when its IoU with another box exceeds 0.9. The authors found that some heavily overlapping duplicate boxes can't be handled by the decoder's self-attention, so they kept this very loose non-maximum suppression.

A removed track query isn't thrown away immediately. It keeps being decoded for another 5 frames, but produces no output during that time. If its score climbs back above 0.4, it's restored as a normal track. This is a short-term re-identification that still relies on the same attention, with no extra re-ID training.

But track queries carry position information. If an object is occluded for too long or moves too far, it can't be found again.

Training

For track queries to learn to follow objects, the model has to be trained on two adjacent frames:

  1. In frame t−1t-1, detection uses object queries only.
  2. Detected objects become track queries and are fed into frame tt along with the object queries.

Each output in frame tt has to be assigned to a ground-truth object or to background. Let Kt−1K_{t-1} and KtK_t be the sets of ground-truth identities in the two frames. The assignment rules are:

  • Kt∩Kt−1K_t \cap K_{t-1} (present in both frames): track queries are matched directly by identity. This is a hard assignment.
  • Kt−1∖KtK_{t-1} \setminus K_t (present in the previous frame, gone in this one): the corresponding track queries have to predict background.
  • Kt∖Kt−1K_t \setminus K_{t-1} (newly appeared): handed to the object queries, which use DETR's Hungarian algorithm to find the lowest-cost matching.

The matching cost combines the class probability and the difference between boxes:

Cmatch=−λcls p^σ(i)(ci)+Cbox(bi,b^σ(i))\mathcal{C}_{match} = -\lambda_{cls}\,\hat{p}_{\sigma(i)}(c_i) + \mathcal{C}_{box}(b_i, \hat{b}_{\sigma(i)})

The box cost combines L1 distance and generalized IoU:

Cbox=λℓ1∥bi−b^σ(i)∥1+λiou Ciou(bi,b^σ(i))\mathcal{C}_{box} = \lambda_{\ell_1}\big\|b_i - \hat{b}_{\sigma(i)}\big\|_1 + \lambda_{iou}\,\mathcal{C}_{iou}(b_i, \hat{b}_{\sigma(i)})

Once assignment is done, the loss for each query is

Lquery={−λclslog⁡p^i(cπ=i)+Lbox,i∈π−λclslog⁡p^i(0),i∉π\mathcal{L}_{query} = \begin{cases} -\lambda_{cls}\log\hat{p}_i(c_{\pi=i}) + \mathcal{L}_{box}, & i \in \pi \\ -\lambda_{cls}\log\hat{p}_i(0), & i \notin \pi \end{cases}

Queries that aren't assigned to anything are all treated as background.

Three augmentations

Trained only on adjacent pairs of frames, the model sees tracking situations that are too simple. The authors add three augmentations:

  • A wider frame gap. Frame t−1t-1 isn't necessarily the immediately preceding frame; it's sampled from a range around tt, simulating camera motion and low frame rates.
  • Missed detections. With probability pFNp_{FN}, some track queries are dropped, so those objects have to be re-detected by object queries in frame tt. This is the key to training the two kinds of queries together.
  • False positives. With probability pFPp_{FP}, a fake track query is added for a given track query, taken from a previous-frame output embedding that was judged to be background and overlaps heavily with it. This lets the model practice removing tracks during occlusion.

Data

Transformers need lots of data, and MOT17 has only 7 training videos. Like CenterTrack, the authors use single images from CrowdHuman to simulate pairs of adjacent frames, applying random scaling and cropping of up to 20% to the same image.

The private-detection model is first trained from scratch on CrowdHuman for 85 epochs, then fine-tuned on a mix of CrowdHuman and MOT17 for 40 epochs, which takes about two days on 7 GPUs with 32GB each.

Discussion

Component ablation

SettingMOTAΔIDF1Δ
Full TrackFormer71.373.4
No CrowdHuman pretraining69.3−2.071.8−1.6
Also no track-query re-ID69.2−0.170.4−1.4
Also no FP augmentation68.4−0.870.0−0.4
Also no frame-gap augmentation64.0−4.459.2−10.8
Also no track queries61.0−3.045.1−14.1

Compiled from Table 3 of the paper, MOT17 training set split into first and second halves, private detections; each row removes one more component from the row above.

This table is cumulative: each row has one fewer component than the row above.

Of the training tricks, the frame-gap augmentation has the biggest effect. Without it, MOTA drops 4.4 points and IDF1 drops 10.8 points. Trained only on immediately adjacent frames, the model overfits quickly and never learns how to follow objects that move a lot.

The last row removes even the track queries. The model is trained only for detection and uses CenterTrack-style greedy matching on center distance instead, and IDF1 falls another 14.1 points. This row stands for the older approach of matching by hand-written rules.

MOT17 and MOT20

Original Table 1. Comparison on the MOT17 and MOT20 test sets, with columns for extra training data, speed, MOTA, IDF1, MT, ML, FP, FN and ID switches. MOT17 public detections: TrackFormer MOTA 62.3, IDF1 57.6, 4,018 ID switches, 7.4 FPS; CenterTrack 60.5, 55.7, 2,540; TMOH 62.1, 62.8. MOT17 private detections: TrackFormer (CrowdHuman) 74.1, 68.0, 2,829 ID switches; CenterTrack 67.8, 64.7; QuasiDense 68.7, 66.3; FairMOT, which uses 6 extra tracking datasets, 73.7, 72.3; GRTU 75.5, 76.9; TLR 76.5, 73.6. MOT20 private detections: TrackFormer 68.6, 65.7; SOTMOT 68.6, 71.4; ReMOT 77.4, 73.1.
Original Table 1, Meinhardt et al., from PDF page 7, CC BY 4.0.

With private detections, TrackFormer reaches MOTA 74.1 and IDF1 68.0 on MOT17. Among methods that use only CrowdHuman as extra data, its MOTA is 5.0 points higher than the runner-up, TraDeS, and 6.3 points higher than CenterTrack. Its FN is only 108,777, more than fifty thousand fewer than CenterTrack, so its recall is much higher.

But the table also has several methods that use 6 extra tracking datasets, such as FairMOT, GRTU and TLR, and all of them have a higher IDF1. On MOT20, its MOTA ties SOTMOT, but it's nearly 9 points below ReMOT, and it trails on IDF1 too.

It runs at 7.4 frames per second, less than half the speed of CenterTrack's 17.7.

How to compare with public detections

TrackFormer isn't a method that takes detection boxes as input and associates them, so it can't use public detections as input directly. The authors use filtering instead: only object queries whose IoU with some public detection box exceeds 0.5 are allowed to start new tracks.

CenterTrack has a similar filter, but it uses center points; a public detection counts if its center falls inside the box. In the appendix, the authors compare the two filters and find that with the center-point version, three public detection sets of very different quality (DPM, FRCNN, SDP) produce almost identical results. In effect, the quality of the detections isn't reflected. The IoU version is stricter and lowers all the scores, but it's closer to the original intent of "using public detections as input."

Under IoU filtering, TrackFormer's MOTA is 62.3, on par with the best online methods; but its 4,018 ID switches are the most in the table.

Adding masks

TrackFormer can also take a segmentation head and do multi-object tracking and segmentation on MOTS20. The authors found that after training jointly with masks, IDF1 goes up by 1.2 points even when only box tracking is evaluated; the mask information helps resolve some ambiguous occlusion cases during training.

Limitations and what came after

TrackFormer is trained only on pairs of adjacent frames, so the temporal information its track queries learn is very short. The authors themselves write in the paper that methods going beyond two-frame training and inference should have a lot of potential.

MOTR, a few months later, did exactly that. It extends training to a whole short clip, passes track queries across many frames, and removes rules like track NMS and re-identification. Also, the TrackFormer numbers cited in the comparison tables of MOTR and the later MOTRv2 are MOTA 65.0 and IDF1 63.9 on MOT17. Those come from an earlier version; only after the final version was trained on adjacent frames simulated from CrowdHuman did they rise to the 74.1 and 68.0 here.

Conclusion

TrackFormer lets one query stand for one object, and once the object is detected, the query follows it from frame to frame. New objects go to object queries, existing ones to track queries, and the two can see each other inside the same decoder. That's how the matching step disappears.

Its numbers don't lead across the board, and it's slow. But it demonstrated another way of writing a tracker, and MOTR and MOTRv2 both built on it afterward.

Two years earlier, Meinhardt and Leal-Taixé used Tracktor to show that you can track very well without training for tracking at all. This time, tracking became something you train for two days on seven GPUs.