Matching Anything by Segmenting Anything
- Publication
- CVPR 2024Conference
- First submitted to arXiv
- arXiv:2406.04221
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingmasa2024,
title = {{Matching Anything by Segmenting Anything}},
author = {Siyuan Li and Lei Ke and Martin Danelljan and Luigi Piccinelli and Mattia Segu and Luc Van Gool and Fisher Yu},
year = {2024},
eprint = {2406.04221},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2406.04221}
}
Around 2023, detection and segmentation got foundation models that "recognize anything." SAM can cut out everything in an image; Detic and Grounding-DINO can detect thousands of categories from text.
Tracking stayed where it was. Methods that learn association almost all have to train on videos with ID annotations, and that kind of data is expensive and narrow: MOT17 has only pedestrians, and BDD100K only eight classes of things on the road. Appearance features learned on pedestrian data stop working once you use them to track animals or cartoon characters.
Siyuan Li, Lei Ke, Fisher Yu and colleagues at ETH Zurich wanted a way to make association "track anything," the way detection can. They use no video at all, just a pile of unannotated photos.
Defining the problem
To learn association, you need to know that "this thing in this frame and that thing in that frame are the same one." That's ID annotation, the most expensive part of tracking data.
People have tried doing without annotation before. Methods like UniTrack use image features learned by self-supervision directly for association. The trouble is that these features are usually trained on data like ImageNet, with "one main subject per image," so what they learn is whole-image similarity. In a street scene with dozens of things in one image, they can't really tell who's who.
The authors set out to solve two things:
- Where the correspondences come from. Without video, how do you know two things are the same one?
- How to attach it to existing models. Detection and segmentation already have strong foundation models, so the association ability should ideally be added on directly, without touching their original abilities.
Solving the problem
One photo, two views

MASA takes a photo and applies two different strong augmentations, for example enlarging, shifting and recoloring it on one side, and shrinking, flipping and mixing another image into it on the other. The two views look very different, but since we made the transformations ourselves, we know which pixel maps to which.
This simulates how the same object changes between two frames of a video: the position changes, the size changes, the lighting changes, and it's still the same thing.
Pixel correspondence alone isn't enough; what's needed is correspondence between "objects." SAM cuts every thing in the photo into its own mask. Pixels in the same mask belong to the same object, so pixel correspondence becomes object correspondence.
Pull the same fish together, push different fish apart
With object correspondence in hand, training is standard contrastive learning. The features of the same object in the two views are a positive pair, to be pulled together; against every other object they're negatives, to be pushed apart:
is cosine similarity, with temperature .
SAM's value here lies in the negatives. In a complex photo, SAM can cut out hundreds of objects, and each one is a negative for all the others. With a temperature this low, the few most similar negatives dominate the loss. Let's run a few numbers.
| Case | Pos. sim. | Neg. sims. | Loss |
|---|---|---|---|
| No look-alikes | 0.80 | 0.30, 0.20, 0.10 | 0.001 |
| One look-alike | 0.80 | 0.75, 0.20, 0.10 | 0.40 |
Worked example for this note.
Swap just one negative for "another fish that looks a lot like it," and the loss grows nearly 400-fold. The model is forced to learn the details that tell similar objects apart. And telling look-alikes apart is exactly the hardest part of tracking.
An adapter behind the foundation model

The learned association ability has to be attachable to any detection or segmentation model. The authors designed a MASA adapter that sits behind a frozen backbone. The backbone stays put, so the original detection and segmentation abilities are unaffected.
The adapter does three things:
- Builds a feature pyramid. Swin Transformer already has multi-scale features, so it connects straight to an FPN; SAM's ViT has only a single scale, so transposed convolutions and pooling create four scales.
- Dynamic feature fusion. Deformable convolutions aggregate information across positions and scales. The authors' reasoning is that to tell different objects apart, the feature at each position had better know what things look like elsewhere too.
- Tracking head. Features are extracted for each object box with RoI-Align and passed through 4 convolutional layers and 1 fully connected layer, outputting an embedding vector for each object.
Training also has an auxiliary task: a detection head that learns to predict the bounding boxes of SAM's masks. This amounts to distilling SAM's knowledge of "what counts as an object" into the adapter's features.
Two ways to use it at inference

When attached to a detector, the detection head used in training is dropped; whatever boxes the detector gives, the adapter produces tracking features for those boxes.
When attached to SAM, the detection head turns out to be useful. To "segment everything," SAM originally has to scatter about a thousand points across the image as prompts and then do tedious post-processing. MASA's detection head gives object boxes directly as prompts, and the authors measured on UVO that at the same recall it's about 10 times faster.
Matching follows the bidirectional softmax of the QDTrack line: the score between an object and a track has to be high in both directions, among "all candidates for this track" and among "all tracks for this object," before they count as a match; this is then averaged with cosine similarity.
The training data is 500,000 photos drawn from SA-1B. Every benchmark uses the same set of weights, never trained on any of the test domains, which makes it zero-shot association.
Discussion
Against methods trained in-domain
To compare association ability alone, on each benchmark the authors use the same detections as the strongest method of the time:
| Benchmark | Rival (in-domain) | Rival AssocA | MASA AssocA |
|---|---|---|---|
| TAO TETA | TETer | 36.7 | 37.6 |
| BDD MOTS | UNINEXT-H | 53.2 | 54.5 |
| BDD MOT | ByteTrack | 51.5 | 52.9 |
TAO compiled from Table 1 of the paper, BDD MOTS from Table 4 of the paper, BDD MOT from Table 5 of the paper, all on validation sets; for MASA, the best variant in each table.
These rivals all used full video annotations in their own domains; MASA never saw a single TAO or BDD image, and its association scores are still slightly higher. The margins are small, but they all point the same way.
On BDD MOT, ByteTrack's mMOTA is 45.5 and MASA's about 44.5. ByteTrack also links low-score boxes into tracks, which favors detection-oriented metrics.
Without the shared detections, letting MASA attached to Detic do its own detection and tracking, its TETA on TAO is 46.3, far above the 34.6 of the other methods in the table. That part comes mainly from Detic's own detection ability; MASA lets it be used directly for tracking.
Unseen categories
The open-vocabulary tracking benchmark allows training only on annotations of common categories and tests rare categories as "novel classes." MASA with Detic gets a novel-class TETA of 40.8 and AssocA of 41.2; OVTrack, designed specifically for this task, gets 32.0 and 33.2.
What each design contributes
The ablations train on raw BDD and COCO photos and test on BDD MOT:
| Setting | AssocA | mIDF1 |
|---|---|---|
| SAM features as is | 32.9 | 37.3 |
| + MASA training and tracking head | 48.5 | 51.7 |
| + dynamic feature fusion | 50.1 | 53.3 |
| + object prior distillation | 51.9 | 54.9 |
Compiled from Table 8 of the paper, BDD MOT validation set.
SAM's original features are optimized for segmentation; used directly to tell individuals apart, they give an AssocA of only 32.9. Add MASA's training and it jumps by 15.6, the biggest single gain in the whole paper.
The other ablations echo the analysis above:
- Proposal source. Replacing SAM with Mask2Former trained on COCO drops TAO AssocA from 34.1 to 29.8. SAM cuts finer and more varied pieces, providing better negatives.
- Number of proposals. Raising the cap on objects per batch from 64 to 256 lifts TAO AssocA from 31.5 to 35.8.
- Augmentation strength. With only basic augmentations like flipping and cropping, BDD mIDF1 is 48.2; adding random affine, MixUp and large-scale jitter raises it to 54.9. The photos have to vary enough to simulate the changes in a video.
Against other self-supervised methods trained on the same BDD and COCO photos, MoCo v2 gets a BDD MOT AssocA of 42.7, DINO only 23.1, and MASA 51.9. Those methods learn whole-image similarity and aren't good at multi-object scenes.
Limitations
In the appendix the authors list two points. First, a foundation model's detections aren't necessarily consistent between adjacent frames; something cut out in this frame may vanish in the next, making the tracking results flicker, and MASA only handles association, so it can't fix detection errors. Second, MASA has no long-term memory, just a queue holding each object's embeddings, so it tends to fail under heavy occlusion.
Conclusion
MASA proves that learning association for tracking doesn't necessarily need video, let alone ID annotations. Transform one photo twice, let SAM upgrade pixel correspondence into object correspondence, and contrastive learning gets plenty of positives and negatives.
The adapter design lets this ability attach to foundation models like SAM, Detic and Grounding-DINO without touching their original detection and segmentation. On TAO and BDD, domains it has never seen, its zero-shot association scores even come out slightly above methods trained in-domain with full annotations.
As for unstable detections and long occlusions, it still leaves those to the foundation model at the front and a simple queue; to track for longer, memory has to be added on separately.