Back to paper library
arXiv 2026CVPR 2026Object trackingPaper notes

FADE

Blackout

Original paper

Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

Paper authors

Halima Bouzidi · Haoyu Liu · Yonatan Gizachew Achamyeleh · Praneetsai Vasu Iddamsetty · Mohammad Abdullah Al Faruque

Publication
CVPR 2026Conference
First submitted to arXiv
arXiv:2604.00452
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingfade2026,
  title = {{Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation}},
  author = {Halima Bouzidi and Haoyu Liu and Yonatan Gizachew Achamyeleh and Praneetsai Vasu Iddamsetty and Mohammad Abdullah Al Faruque},
  year = {2026},
  eprint = {2604.00452},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.00452}
}
On this page

Since MOTR, query-based tracking has kept evolving: MOTRv2 bolted on an external detector, MeMOTR added long-term memory, CO-MOT changed the label assignment, and SambaMOTR used a state-space model to remember the whole history. This line relies more and more on "memory," with each query carrying an object's information forward from the moment it appeared up to now.

Memory makes tracking more stable, but what if someone deliberately breaks the memory?

That's exactly what Halima Bouzidi, Mohammad Al Faruque and colleagues at the University of California, Irvine set out to do. This is an attack paper, aimed at finding the structural weaknesses of this kind of tracker. Past adversarial attacks on trackers were almost all designed for detect-then-associate methods; this paper is the first to target query-based trackers specifically.

Defining the problem

Old attacks can't reach new trackers

Attacks on detect-then-associate trackers fall roughly into three kinds:

  1. Disrupting detection. Make real objects go undetected, or create large numbers of fake detection boxes. Daedalus is the latter kind.
  2. Disrupting association. Attacks like Hijacking and F&F pull detection boxes away from the Kalman filter's predictions; once the gap passes a threshold, the boxes can't be matched, or get matched to someone else.
  3. Disrupting appearance features. Alter the ReID embedding vectors directly so that similarities come out wrong.

But query-based trackers have no fixed-threshold NMS, no Kalman filter and no separate bank of ReID features. Duplicate boxes suppress one another through attention, and motion is learned through query propagation. The components these attacks target simply don't exist in the new architecture.

Three weaknesses of the new architecture

The authors argue that query-based trackers have three structural weaknesses:

  1. The number of queries is fixed. Track queries have a cap; every seat a fake track takes is one fewer for a real track. It's zero-sum.
  2. State is passed on recursively. Each query's state in this frame depends on its state in the previous frame. Once one frame is corrupted, the error is passed along with it.
  3. Memory amplifies the damage. MeMOTR's long-term memory and SambaMOTR's state-space model were meant to remember a longer history, and corrupted memory sticks around longer too.

By contrast, detect-then-associate methods redo association from scratch every frame, which amounts to a built-in reset. For query-based trackers, the same property cuts both ways: temporal smoothing can filter out single-frame noise, and temporal dependence also lets errors carry on.

Solving the problem

Original Figure 2. The FADE attack pipeline. (a) The perturbation is optimized with projected gradient descent (PGD); it can be digital pixel noise or a simulated physical attack: motion blur caused by acoustic waves, or colored horizontal stripes caused by electromagnetic interference. (b) The attacked tracker: image encoder, Transformer decoder, query updater and query state memory. The tracker's outputs and hidden states are used to compute the attack loss, and the gradient flows back to (a). (c) Illustrations of the two attacks: query flooding fills the queries with fake tracks so the real tracks get none; memory corruption breaks real tracks so that they come back with new IDs.
Original Figure 2, Bouzidi et al., from PDF page 4, arXiv license, scholarly quotation.

FADE has two attack strategies, matching the weaknesses above. Both add a perturbation to the image and use gradient optimization to make a particular loss larger.

Attack one: query flooding

The first attack is called TQF (temporal query flooding), and it targets the fixed number of queries.

Suppose the tracker can maintain at most 30 tracks at once, and there are 21 people in the frame, roughly the MOT17 average. If the attack can create 12 fake tracks that persist, total demand becomes 33, and at least 3 real people get no query.

Numbers assumed for this note; the paper doesn't give a specific query limit.

Creating fake detections alone isn't enough: query-based trackers filter out discontinuous fake boxes within a few frames. So TQF is made of three parts:

  1. Flooding. Make every currently idle query output a confidence close to 1.
  2. Disguising the cost. Make the matching cost between these fake queries and real objects very low, luring the Hungarian matching into assigning real people's identities to the fake queries.
  3. Stealing identities. Make the fake queries' hidden states as similar as possible to the hidden states of real tracks in the previous frame. The tracker then takes the fake query as the continuation of some known object, and the fake query takes over in its place, history and all.

Attack two: memory corruption

The second attack is called TMC (temporal memory corruption). Rather than creating new tracks, it makes existing tracks lose their memory. It has two parts:

  1. Cutting the temporal link. Make each track's hidden state in this frame as unlike its hidden state in the previous frame as possible. What the query updater sees then looks like a new object with no connection to the past.
  2. Erasing features. Push the embedding vectors of tracks that have already been matched toward zero. A vector close to zero has a dot product close to zero with anything, so it becomes very hard to recognize again later.

From digital to physical

The digital attack adds noise directly to the pixels, limited to at most 8/255 per pixel, and attacks only one frame.

The authors also simulate two physical attacks, both sensor-spoofing techniques that earlier research verified on real cameras:

  1. Acoustic injection. Sound waves at specific frequencies make the camera's image-stabilization components resonate, producing motion blur in the image.
  2. Electromagnetic interference. Interfering with the sensor's analog-to-digital conversion makes colored horizontal stripes appear in the image.

Both effects are written as differentiable models, where the only adjustable parameters are the blur's amplitude and phase, and the stripes' position, color and width, each limited to ranges calibrated by earlier measurements. The optimization looks for "the most damaging set of parameters within what's physically achievable." The physical attacks last 3 frames.

Discussion

Effect of the digital attacks

TrackerCleanDaedalusHijackingTQFTMC
MOTR58.6345.8546.0845.9045.89
MOTRv259.9639.2156.3139.2946.76
MeMOTR67.3542.4155.9241.4141.56
SambaMOTR62.9154.4057.4045.5348.04
CO-MOT58.1640.0151.6837.2641.73

Compiled from Table 1 of the paper, MOT17, HOTA.

Attacking just one frame, under the most effective attack for each, the five trackers lose 13 to 26 points of HOTA. Hijacking, designed against the Kalman filter, costs MOTRv2 only 3.65 points, far from FADE's 20.67.

But the table also shows that Daedalus, which simply creates fake detections, is almost as effective as FADE on MOTR, MOTRv2 and CO-MOT. Where FADE really pulls ahead is on trackers with memory mechanisms: on SambaMOTR, Daedalus only brings HOTA down to 54.40, while TQF brings it down to 45.53.

The contrast is sharper on MOT20. MeMOTR scores HOTA 69.61 clean and still 62.51 under Daedalus; under TMC it's left with only 37.70, and ID switches increase more than tenfold. The authors' reading is that a tracker with memory can filter out noise that doesn't persist, but once the memory itself is contaminated, it's helpless.

One frame of attack, fifteen frames of blackout

Original Figure 3. MeMOTR under TQF attack on a MOT17 video. Frame 9 is clean, and everyone is tracked; frame 10 is attacked, and a large number of fake boxes pop up; in frames 11 and 20 the image is clean again, yet there are no tracks at all; only at frame 25 does the tracker re-establish tracks, and by then everyone has a new ID.
Original Figure 3, Bouzidi et al., from PDF page 5, arXiv license, scholarly quotation.

This figure best shows the nature of this kind of attack. The attack happens only in frame 10, and every frame after it is clean. Yet MeMOTR drops all its tracks in frame 11, real and fake alike, and doesn't start tracking again until 15 frames later. When it does, everyone has a new ID.

With an ordinary adversarial attack, the model goes back to normal once the perturbation is removed. Here the damage stays in memory and keeps going after the attack is over.

Physical attacks

Original Figure 4. TMC under simulated physical attacks. The top row is acoustic injection: frame 10 shows motion blur, but most pedestrians are still clearly visible to the human eye. The bottom row is electromagnetic interference: a few colored horizontal stripes appear in frame 10. In both cases the image is clean again from frame 11 on, yet the tracker can't reconnect people to their original IDs and still hasn't recovered by frame 25.
Original Figure 4, Bouzidi et al., from PDF page 8, arXiv license, scholarly quotation.

The simulated physical attacks are weaker than the digital ones, but still effective. On MOT17, MOTR drops from 58.63 to 42.81 under electromagnetic interference, and SambaMOTR from 62.91 to 48.97 under acoustic blur.

MOTRv2, with its external detector, and CO-MOT, with its stronger detection, hold up better, losing roughly 5 to 7 points of HOTA. Even so, their association metric, AssA, still drops by about 8 points. The detector blocks most of the blur and stripes, but the little that slips through is still enough to contaminate the query states downstream.

The more people, the more effective the attack. Under the acoustic attack, MOTR loses 13.5 points on MOT17, and 18.9 points on MOT20, with about 150 people per frame, where AssA falls all the way from 75.89 to 39.72.

Which part matters most

Part removedCO-MOTMeMOTRSambaMOTR
Full TMC41.7341.5648.04
No link cutting53.2751.7961.63
No feature erasure41.9141.9052.31

Compiled from Table 5 of the paper, MOT17, HOTA after attack; higher numbers mean a weaker attack.

Cutting the temporal link is the core of TMC. Remove it, and SambaMOTR's HOTA comes almost back to the clean level (61.63 vs. 62.91). Erasing features is a supporting part; removing it has much less effect.

The three parts of TQF matter differently for different trackers. SambaMOTR is most sensitive to "disguising the cost": without that term, its HOTA after attack recovers from 45.53 to 53.14, while removing "stealing identities" actually makes the attack slightly stronger. The authors speculate that SambaMOTR's state-space model filters out this kind of imitated identity, so to fool it, you have to go after the matching cost. CO-MOT is the opposite, more sensitive to identity stealing.

Defenses and transfer

In the appendix, the authors try three common input preprocessing steps: color jitter, spatial smoothing and added Gaussian noise.

They do almost nothing for MOTR: after the TMC attack's 45.89, the best they manage is 46.38. MeMOTR recovers best against TQF, from 41.41 back to 63.39, close to its clean 67.35. SambaMOTR is hard to rescue: 48.04 after TMC, and only 50.97 even with defenses. The differences among the three defenses are mostly under 0.5 points.

In the black-box setting, the attacker can't access the target model and has to generate the perturbation with another model. The effect then weakens noticeably: for example, attacking MeMOTR with perturbations generated by other trackers still leaves HOTA at 55 to 60, and only perturbations generated with MeMOTR itself bring it down to 29.10. The authors admit that cross-architecture black-box attacks remain an open problem.

Limitations

Beyond black-box transfer, the authors also point out that the physical attacks were verified only in simulation; although the parameters come from real camera calibrations, they were never actually tested on a real camera and tracking system. Optimizing the perturbation takes about 3 seconds per frame on an RTX 3090, and the digital-attack experiments also assume the attacker has access to the model's gradients.

Conclusion

What FADE points to is a structural trade-off. By passing state and memory forward frame after frame, query-based trackers learned to handle occlusion and long-term association; the same mechanism also lets one frame of interference turn into a dozen-plus frames of blackout. And the fixed number of queries lets fake tracks crowd out real ones.

Judging by the experiments, the stronger a tracker's memory, the more resistant it is to simple noise, and the more fragile it is to attacks aimed specifically at memory. The defense experiments in the appendix also show that input-side preprocessing alone can't stop this kind of attack, especially for SambaMOTR with its state-space model.