Back to paper library
arXiv 2024CVPR 2024Object trackingPaper notes

MASA

Match the Pairs

Original paper

Matching Anything by Segmenting Anything

Paper authors

Siyuan Li · Lei Ke · Martin Danelljan …

Show all 7 authors

Siyuan Li · Lei Ke · Martin Danelljan · Luigi Piccinelli · Mattia Segu · Luc Van Gool · Fisher Yu

Publication
CVPR 2024Conference
First submitted to arXiv
arXiv:2406.04221
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingmasa2024,
  title = {{Matching Anything by Segmenting Anything}},
  author = {Siyuan Li and Lei Ke and Martin Danelljan and Luigi Piccinelli and Mattia Segu and Luc Van Gool and Fisher Yu},
  year = {2024},
  eprint = {2406.04221},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.04221}
}
On this page

Around 2023, detection and segmentation got foundation models that "recognize anything." SAM can cut out everything in an image; Detic and Grounding-DINO can detect thousands of categories from text.

Tracking stayed where it was. Methods that learn association almost all have to train on videos with ID annotations, and that kind of data is expensive and narrow: MOT17 has only pedestrians, and BDD100K only eight classes of things on the road. Appearance features learned on pedestrian data stop working once you use them to track animals or cartoon characters.

Siyuan Li, Lei Ke, Fisher Yu and colleagues at ETH Zurich wanted a way to make association "track anything," the way detection can. They use no video at all, just a pile of unannotated photos.

Defining the problem

To learn association, you need to know that "this thing in this frame and that thing in that frame are the same one." That's ID annotation, the most expensive part of tracking data.

People have tried doing without annotation before. Methods like UniTrack use image features learned by self-supervision directly for association. The trouble is that these features are usually trained on data like ImageNet, with "one main subject per image," so what they learn is whole-image similarity. In a street scene with dozens of things in one image, they can't really tell who's who.

The authors set out to solve two things:

  1. Where the correspondences come from. Without video, how do you know two things are the same one?
  2. How to attach it to existing models. Detection and segmentation already have strong foundation models, so the association ability should ideally be added on directly, without touching their original abilities.

Solving the problem

One photo, two views

Original Figure 1. MASA's training pipeline. A photo is drawn from unannotated images across all sorts of domains and given two different strong augmentations, producing two views whose pixel correspondence is known (red dashed lines). SAM cuts the photo into individual instances, so the pixel correspondence becomes an instance correspondence (green dashed lines). At the bottom is dense similarity learning: the same fish across the two views is a positive pair (green lines), and any two different fish are negatives (red lines).
Original Figure 1, Li et al., from PDF page 1, CC BY 4.0.

MASA takes a photo and applies two different strong augmentations, for example enlarging, shifting and recoloring it on one side, and shrinking, flipping and mixing another image into it on the other. The two views look very different, but since we made the transformations ourselves, we know which pixel maps to which.

This simulates how the same object changes between two frames of a video: the position changes, the size changes, the lighting changes, and it's still the same thing.

Pixel correspondence alone isn't enough; what's needed is correspondence between "objects." SAM cuts every thing in the photo into its own mask. Pixels in the same mask belong to the same object, so pixel correspondence becomes object correspondence.

Pull the same fish together, push different fish apart

With object correspondence in hand, training is standard contrastive learning. The features of the same object in the two views are a positive pair, to be pulled together; against every other object they're negatives, to be pushed apart:

LC=−∑qlog⁡s(q,q+)s(q,q+)+∑q−s(q,q−)s(a,b)=e sim(a,b)/τ\begin{aligned} \mathcal{L}_C &= -\sum_{q}\log\frac{s(q,q^+)}{s(q,q^+) + \sum_{q^-} s(q,q^-)} \\ s(a,b) &= e^{\,\text{sim}(a,b)/\tau} \end{aligned}

sim\text{sim} is cosine similarity, with temperature τ=0.07\tau = 0.07.

SAM's value here lies in the negatives. In a complex photo, SAM can cut out hundreds of objects, and each one is a negative for all the others. With a temperature this low, the few most similar negatives dominate the loss. Let's run a few numbers.

CasePos. sim.Neg. sims.Loss
No look-alikes0.800.30, 0.20, 0.100.001
One look-alike0.800.75, 0.20, 0.100.40

Worked example for this note.

Swap just one negative for "another fish that looks a lot like it," and the loss grows nearly 400-fold. The model is forced to learn the details that tell similar objects apart. And telling look-alikes apart is exactly the hardest part of tracking.

An adapter behind the foundation model

Original Figure 2. MASA's training architecture. Each of the two views passes through a frozen backbone to get multi-scale features; the MASA adapter first processes these features with dynamic feature fusion, then splits into a detection head and a tracking head. The detection head learns the object boxes cut out by SAM (object prior distillation), and the tracking head's outputs undergo instance similarity learning between the two views; the heads for the two views share weights.
Original Figure 2, Li et al., from PDF page 4, CC BY 4.0.

The learned association ability has to be attachable to any detection or segmentation model. The authors designed a MASA adapter that sits behind a frozen backbone. The backbone stays put, so the original detection and segmentation abilities are unaffected.

The adapter does three things:

  1. Builds a feature pyramid. Swin Transformer already has multi-scale features, so it connects straight to an FPN; SAM's ViT has only a single scale, so transposed convolutions and pooling create four scales.
  2. Dynamic feature fusion. Deformable convolutions aggregate information across positions and scales. The authors' reasoning is that to tell different objects apart, the feature at each position had better know what things look like elsewhere too.
  3. Tracking head. Features are extracted for each object box with RoI-Align and passed through 4 convolutional layers and 1 fully connected layer, outputting an embedding vector for each object.

Training also has an auxiliary task: a detection head that learns to predict the bounding boxes of SAM's masks. This amounts to distilling SAM's knowledge of "what counts as an object" into the adapter's features.

Two ways to use it at inference

Original Figure 3. Two inference modes. (a) With a detector (for example, Detic): MASA's detection head is dropped, the detector's boxes are used to extract tracking features, and these are matched against past tracks. (b) With SAM: MASA's detection head is kept, and its predicted boxes serve as prompts both for SAM's mask decoder and for the tracking features, completing segmentation and tracking in one pass.
Original Figure 3, Li et al., from PDF page 5, CC BY 4.0.

When attached to a detector, the detection head used in training is dropped; whatever boxes the detector gives, the adapter produces tracking features for those boxes.

When attached to SAM, the detection head turns out to be useful. To "segment everything," SAM originally has to scatter about a thousand points across the image as prompts and then do tedious post-processing. MASA's detection head gives object boxes directly as prompts, and the authors measured on UVO that at the same recall it's about 10 times faster.

Matching follows the bidirectional softmax of the QDTrack line: the score between an object and a track has to be high in both directions, among "all candidates for this track" and among "all tracks for this object," before they count as a match; this is then averaged with cosine similarity.

The training data is 500,000 photos drawn from SA-1B. Every benchmark uses the same set of weights, never trained on any of the test domains, which makes it zero-shot association.

Discussion

Against methods trained in-domain

To compare association ability alone, on each benchmark the authors use the same detections as the strongest method of the time:

BenchmarkRival (in-domain)Rival AssocAMASA AssocA
TAO TETATETer36.737.6
BDD MOTSUNINEXT-H53.254.5
BDD MOTByteTrack51.552.9

TAO compiled from Table 1 of the paper, BDD MOTS from Table 4 of the paper, BDD MOT from Table 5 of the paper, all on validation sets; for MASA, the best variant in each table.

These rivals all used full video annotations in their own domains; MASA never saw a single TAO or BDD image, and its association scores are still slightly higher. The margins are small, but they all point the same way.

On BDD MOT, ByteTrack's mMOTA is 45.5 and MASA's about 44.5. ByteTrack also links low-score boxes into tracks, which favors detection-oriented metrics.

Without the shared detections, letting MASA attached to Detic do its own detection and tracking, its TETA on TAO is 46.3, far above the 34.6 of the other methods in the table. That part comes mainly from Detic's own detection ability; MASA lets it be used directly for tracking.

Unseen categories

The open-vocabulary tracking benchmark allows training only on annotations of common categories and tests rare categories as "novel classes." MASA with Detic gets a novel-class TETA of 40.8 and AssocA of 41.2; OVTrack, designed specifically for this task, gets 32.0 and 33.2.

What each design contributes

The ablations train on raw BDD and COCO photos and test on BDD MOT:

SettingAssocAmIDF1
SAM features as is32.937.3
+ MASA training and tracking head48.551.7
+ dynamic feature fusion50.153.3
+ object prior distillation51.954.9

Compiled from Table 8 of the paper, BDD MOT validation set.

SAM's original features are optimized for segmentation; used directly to tell individuals apart, they give an AssocA of only 32.9. Add MASA's training and it jumps by 15.6, the biggest single gain in the whole paper.

The other ablations echo the analysis above:

  1. Proposal source. Replacing SAM with Mask2Former trained on COCO drops TAO AssocA from 34.1 to 29.8. SAM cuts finer and more varied pieces, providing better negatives.
  2. Number of proposals. Raising the cap on objects per batch from 64 to 256 lifts TAO AssocA from 31.5 to 35.8.
  3. Augmentation strength. With only basic augmentations like flipping and cropping, BDD mIDF1 is 48.2; adding random affine, MixUp and large-scale jitter raises it to 54.9. The photos have to vary enough to simulate the changes in a video.

Against other self-supervised methods trained on the same BDD and COCO photos, MoCo v2 gets a BDD MOT AssocA of 42.7, DINO only 23.1, and MASA 51.9. Those methods learn whole-image similarity and aren't good at multi-object scenes.

Limitations

In the appendix the authors list two points. First, a foundation model's detections aren't necessarily consistent between adjacent frames; something cut out in this frame may vanish in the next, making the tracking results flicker, and MASA only handles association, so it can't fix detection errors. Second, MASA has no long-term memory, just a queue holding each object's embeddings, so it tends to fail under heavy occlusion.

Conclusion

MASA proves that learning association for tracking doesn't necessarily need video, let alone ID annotations. Transform one photo twice, let SAM upgrade pixel correspondence into object correspondence, and contrastive learning gets plenty of positives and negatives.

The adapter design lets this ability attach to foundation models like SAM, Detic and Grounding-DINO without touching their original detection and segmentation. On TAO and BDD, domains it has never seen, its zero-shot association scores even come out slightly above methods trained in-domain with full annotations.

As for unstable detections and long occlusions, it still leaves those to the foundation model at the front and a simple queue; to track for longer, memory has to be added on separately.