MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking
- Publication
- ICCV 2023Conference
- First submitted to arXiv
- arXiv:2307.15700
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingmemotr2023,
title = {{MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking}},
author = {Ruopeng Gao and Limin Wang},
year = {2023},
eprint = {2307.15700},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2307.15700}
}
MOTR represents each track with a track query. In every frame, that query passes through the decoder, gets updated, and is handed on to the next frame. An object's identity is kept alive by this chain of hand-offs.
The problem is that each frame's query only sees the previous frame. When a dancer is hidden for a few frames, turns around, or brushes past someone dressed the same, this frame's output can become unreliable, and the next frame's query comes straight from that unreliable output.
Ruopeng Gao and Limin Wang (also the corresponding author of SportsMOT) of Nanjing University wanted track queries to remember a bit more of the past. They give each track a long-term memory that updates very slowly and serves to steady the track query. That's MeMOTR.
Defining the problem
The authors point out two problems with query-based trackers like MOTR.
Only adjacent frames are used. Whether they fuse queries or fuse features, most methods explicitly use information from only two adjacent frames. The track query keeps getting updated, but it has no stable long-term representation. If the same person's query changes too much over a few frames, association easily goes wrong. The earlier MeMOT stored past features in a very large memory bank, but the storage cost is high.
Detect queries and track queries aren't on the same level. DETR's learnable object queries act more like learnable anchor boxes, with little semantic meaning of their own. Track queries come from the previous frame's output and already know what class they represent and where they are. Throwing these two kinds of queries into the same decoder together from the start leaves a semantic gap between them.
Solving the problem

The overall architecture is a lot like MOTR's, with a ResNet-50 backbone plus a Transformer encoder extracting image features and a decoder handling detection and tracking at the same time. What differs is that the decoder is split into two parts, each track gets an extra long-term memory, and a temporal interaction module produces the next frame's track embeddings. The detection framework is DAB-Deformable-DETR, and the authors also provide a version with the standard Deformable-DETR, for a fair comparison with MOTR.
Give the detect queries a rough idea first
The original decoder has 6 layers. MeMOTR takes the first layer out as a detection decoder that handles only the detect queries. After this layer, the detect queries have roughly located objects and their classes, becoming detect embeddings that carry semantics.
Then the detect embeddings and the track embeddings are concatenated and fed into the joint decoder made of the remaining 5 layers. By now both kinds of queries "know what they're looking for," so they start from a more consistent place.
Long-term memory
Each newly appearing track initializes a long-term memory with its current output. After that, every frame updates it like this:
is this track's decoder output at frame . This is an exponentially weighted moving average, with .
0.01 is a very small number. The authors' assumption is that an object deforms and moves only slightly over a few consecutive frames, so its semantic features should change slowly, and the memory should update smoothly.
Let's work out how long it remembers. A frame's output enters the memory with weight , and after that it's multiplied by 0.99 every frame. , so after about 69 frames, that frame's influence is down by half. DanceTrack videos run at 20 frames per second, so that's roughly three and a half seconds. Overall, this memory is roughly an average over the last 100 frames, 5 seconds.
When a dancer turns around or is hidden for two or three seconds, the noise from those few frames hardly pulls this memory off course.
The temporal interaction module
How does the long-term memory get used for tracking in the next frame? Through the temporal interaction module.

It works in two steps.
Step one: fuse two adjacent frames. This frame's output may be unreliable because of occlusion or blur, so a small network first produces channel-wise weights for it:
The weights are multiplied with this frame's output, which is then concatenated with the previous frame's output and passed through a two-layer MLP to get the fused . The previous frame's output isn't weighted, because the fact that it survived means its confidence score already passed the threshold, so it's relatively reliable.
Step two: let tracks compare themselves with one another. With several similar-looking people in the same frame, the authors want each track's representation to stand further apart from the others. So a multi-head attention, called the memory-attention layer, lets all the tracks interact, with the fused as the query (Q), the long-term memory as the key (K) and this frame's output as the value (V).
The attention output is then added to the long-term memory and passed through a feed-forward network, giving the next frame's track embedding . Because the slowly changing long-term memory is added in, the track embedding is also less likely to change abruptly between two adjacent frames.
When to update
Inference uses three thresholds, all set to 0.5:
- A detection whose confidence exceeds becomes a new track.
- A tracked object whose confidence falls below isn't deleted right away; it's marked as temporarily lost, and only deleted if it can't be recovered for consecutive frames. is 30 frames on DanceTrack and 15 frames on MOT17.
- Only when the confidence exceeds are the track embedding and long-term memory updated with the temporal interaction module's outputs and ; otherwise the old and are kept.
For the frames where someone is blocked and their confidence drops, their memory is frozen for the moment, so unreliable outputs don't get written into it.
Training follows MOTR's approach of feeding in one video clip at a time, with the clips gradually lengthened. On DanceTrack, it trains for 18 epochs, with clips starting at 2 frames and lengthened to 3, 4 and 5 frames at epochs 6, 10 and 14 respectively.
Discussion
DanceTrack
The authors evaluate mainly on DanceTrack, because its association problem is the hardest.
| Method | HOTA | DetA | AssA | IDF1 |
|---|---|---|---|---|
| ByteTrack | 47.7 | 71.0 | 32.1 | 53.9 |
| MOTR | 54.2 | 73.5 | 40.2 | 51.5 |
| OC-SORT | 55.1 | 80.3 | 38.3 | 54.6 |
| C-BIoU | 60.6 | 81.3 | 45.4 | 61.6 |
| MeMOTR (Deformable-DETR) | 63.4 | 77.0 | 52.3 | 65.5 |
| MeMOTR | 68.5 | 80.5 | 58.4 | 71.2 |
| MOTRv2 (extra data) | 69.9 | 83.0 | 59.0 | 71.7 |
Compiled from Table 1 of the paper, DanceTrack test set.
MeMOTR's HOTA is 7.9 points above C-BIoU, the previous best method without extra data, and its AssA is 13.0 points higher. Even switched back to the same standard Deformable-DETR as MOTR, its HOTA is 63.4, more than 9 points above MOTR.
MOTRv2, which uses extra data and also borrows YOLOX detections, is still slightly higher. But MeMOTR's AssA is only 0.6 behind, so the gap is mainly in detection.
From MOTR to MeMOTR, step by step
In the appendix, the authors list each change in one table, adding them up starting from MOTR:
| Step | Val HOTA | Test HOTA |
|---|---|---|
| MOTR | 51.7 | 54.2 |
| + long-term memory and temporal interaction module | 56.5 | 62.5 |
| + detection decoder (1 + 5 layers) | 61.0 | 63.4 |
| Switch to DAB-Deformable-DETR (decoder not split) | 62.1 | 65.9 |
| + detection decoder (1 + 5 layers) | 63.9 | 68.5 |
| + YOLOX proposals | 66.8 | 70.0 |
Compiled from Table 8 of the paper, DanceTrack. Except for MOTR in the first row, all rows drop MOTR's two training tricks (randomly removing track queries and inserting false-positive queries).
The biggest step is the first one. Adding just the long-term memory and the temporal interaction module makes test-set HOTA jump from 54.2 to 62.5, 8.3 points more. Switching to DAB-Deformable-DETR helps a lot too; the authors argue that an anchor-based position prior suits track boxes, which move smoothly over time. The last row follows MOTRv2 and adds YOLOX proposals, bringing HOTA to 70.0, but then it's no longer a fully end-to-end method.
Ablating each part
The ablations are all done on the DanceTrack validation set, because its training data is about 10 times the size of MOT17's, so overfitting is less of a risk.
- One layer is best for the detection decoder. Not splitting gives 62.1, splitting off 1 layer gives 63.9, with AssA going from 52.2 to 55.0; splitting off 2 layers actually drops it to 63.2, because the joint decoder loses a layer and the track embeddings get corrected one time fewer.
- Adjacent-frame fusion needs both pieces together. Adding only the channel-wise weights, or only the previous frame's output, gains just 0.1 to 0.4 HOTA either way; using both reaches 63.9, 1.6 above using neither. The authors' explanation is that suppressing unreliable channels without the previous frame to fill in the information amounts to simply throwing information away.
- The memory-attention layer needs the long-term memory. Without long-term memory, adding the memory-attention layer actually makes things worse, with AssA falling from 52.1 to 50.7. With long-term memory, adding the attention layer raises AssA from 52.9 to 55.0. The authors argue that without memory, track embeddings are unstable to begin with, and having them compare against one another only amplifies the noise.
- is best. At 0.005, 0.02 and 0.04, HOTA is 62.6, 63.2 and 63.5 respectively. The authors also note that may need adjusting per dataset; for example, scenes where objects deform a lot may need faster updates.

This figure illustrates point 3. In (b), with only attention added, the points of different tracks are even more jumbled than in (a); in (d), with both memory and attention, each track's embeddings over the 50 frames gather into a single clump, and the clumps are the most clearly separated.
The hardest videos
The authors pick the 8 videos in the DanceTrack validation set where MOTR's AssA is lowest and compare on those alone. MOTR's average AssA on these 8 is only 23.6; MeMOTR raises it to 35.5.
That's a clear improvement, but still far below MeMOTR's overall 58.4. The authors themselves admit that the association problem is far from solved.
MOT17 and other datasets
On MOT17, MeMOTR's HOTA is 58.8, the highest among Transformer-based methods, and its AssA of 58.4 and IDF1 of 71.5 both beat the earlier MeMOT; but its detection DetA is only 59.6, far below ByteTrack with YOLOX (64.5), and its HOTA also trails ByteTrack's 63.1.
The authors give two reasons. First, MOT17's pedestrians are small and dense, which DETR-style detection already struggles with, and the detect queries are further suppressed by the track queries in self-attention. Second, MOT17's training set has only about 5,000 frames; even with pseudo-videos made from CrowdHuman added, the model's HOTA is about 90 on the training set but only 58.8 on the test set, which is very severe overfitting.
On the multi-class BDD100K validation set, measured by TETA, MeMOTR scores 53.6, above TETer's 50.8 and MOTR's 50.7.
Version 3 adds results on SportsMOT. Using only the SportsMOT training set, MeMOTR's HOTA is 70.0 and its AssA 59.1, above OC-SORT (68.1), which was also trained only on the training set and rerun with its official code. The authors deliberately don't use extra data, on the grounds that the SportsMOT training set already has over 25,000 frames, and that shifting static images into fake videos departs from real tracking scenarios, which is bad for end-to-end methods that learn directly from the data.
Conclusion
MeMOTR gives each track a slowly updating long-term memory, injects it into the next frame's track embedding through the temporal interaction module, and also splits off a detection decoder layer so detect queries and track queries start from the same place. The cumulative table in the appendix shows that the biggest step is the long-term memory itself, which alone adds more than 8 points to MOTR on the DanceTrack test set.
Its weaknesses are inherited from MOTR, too. Detection is still the weak spot; on crowded MOT17 it loses to methods using YOLOX, and on small datasets it overfits easily. The authors suggest trying synthetic data next.
This team's next step was MOTIP in 2024, which recasts identity association as a classification problem.
A memory that changes only 1% per frame sounds sluggish, but picking out who's who among a group of dancers in identical outfits depends on exactly that sluggishness.