Back to paper library
arXiv 2020NeurIPS 2020Contrastive learningPaper notes

SwAV

Swapping Cluster Predictions

Original paper

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Paper authors

Mathilde Caron · Ishan Misra · Julien Mairal · Priya Goyal · Piotr Bojanowski · Armand Joulin

Publication
NeurIPS 2020Conference
First submitted to arXiv
arXiv:2006.09882
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{contrastivelearningswav2020,
  title = {{Unsupervised Learning of Visual Features by Contrasting Cluster Assignments}},
  author = {Mathilde Caron and Ishan Misra and Julien Mairal and Priya Goyal and Piotr Bojanowski and Armand Joulin},
  year = {2020},
  eprint = {2006.09882},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.09882}
}
On this page

There has been some debate between the MoCo and SimCLR series for a while.

Previously, we focused on comparing positive and negative samples in pairs. This time, let's explore a different approach: how about trying clustering?

Problem Definition

The structure of contrastive learning has largely been established, with several main ingredients:

  1. Using contrastive loss: InfoNCE-style losses, including SimCLR’s NT-Xent, is one commonly used loss function.
  2. Applying image transformations: The features of the same image are pulled together, while features of different images are pushed apart.
  3. A large number of negative samples: More negatives can help, depending on the data and training setup.

Of course, there are many variations in the details, such as using momentum encoders or memory banks.

The authors of this paper believe: "The large computational resources are still a significant issue!"

What if we change the approach and stop focusing solely on contrastive loss, instead turning it into a "prediction" problem?

We have the model "predict" the cluster assignments of another view, rather than directly comparing the features. The advantage here is that we no longer need to compute all pairwise comparisons but can achieve the same effect through cluster assignments.

Solving the Problem

Model Architecture

Looking at this diagram, you’ve essentially understood the paper.

First, the left side shows the currently widely accepted contrastive learning architecture:

In the diagram, you can see that an image XX undergoes various transformations (transformations t∼Tt \sim T) to produce two augmented views (e.g., X1,X2X_1, X_2).

These two augmented views are each passed through a convolutional network (or any feature extraction network) fθf_\theta to output feature vectors (Z1,Z2Z_1, Z_2).

In contrastive instance learning, the method "directly compares these two feature vectors", such as calculating their similarity (similar to methods like SimCLR, MoCo, etc.). The model aims to bring features from the same image closer together in feature space, while features from different images should be pushed farther apart.


The right side shows the method proposed in this paper, let’s zoom in:

In SwAV, the authors first assign each feature (e.g., Z1,Z2Z_1, Z_2) to a set of trainable "prototype vectors" (hereafter referred to as prototypes), obtaining corresponding "codes" Q1,Q2Q_1, Q_2.

This step is represented by "Prototypes C" and "Codes" in the diagram.

Next, the core of SwAV is the "Swapped Prediction", meaning:

  1. Use the code Q1Q_1 corresponding to the feature of view 1 to guide what the feature of view 2 should learn.
  2. Use the code Q2Q_2 corresponding to the feature of view 2 to guide what the feature of view 1 should learn.

Thus, SwAV does not directly compare Z1Z_1 and Z2Z_2. Instead, the features of one view predict the other view's assignment to the prototypes, encouraging consistent assignments across views of the same image.

Since prototypes are a set of learnable vectors, they can be seen as "cluster centers" used to map feature vectors to "codes," forming a clustering effect. During training, the model simultaneously learns these prototypes, placing them in positions in feature space that better distinguish different images.

Online Clustering

An important feature of SwAV is "Online Clustering."

Let’s take a look at how the authors calculate the "code" for each image online and update the prototypes:

For each image xnx_n, a transformation tt is randomly sampled from a set of transformations TT (e.g., random cropping, horizontal flipping, color jittering), generating an augmented image xntx_{nt}. Then, a convolutional neural network is used to extract feature vectors, and the feature vectors are projected onto the unit sphere, obtaining zntz_{nt}, i.e., normalizing the length of the output vector to 1.

At this point, there is not much difference from previous contrastive learning papers.

Finally, zntz_{nt} is mapped to KK trainable "prototypes" {c1,…,cK}\{ c_1, \dots, c_K \}, resulting in a "code" qntq_{nt}.

After obtaining the cluster code for each image, the next step is to perform the swapped prediction:

Two augmented views (t,s)(t, s) correspond to the same image. Let (zt,qt)(z_t, q_t) represent the feature and code for view tt, and (zs,qs)(z_s, q_s) represent the feature and code for view ss.

  • The authors want to use ztz_t to "predict" qsq_s, and zsz_s to "predict" qtq_t.
  • This differs from directly comparing feature vectors in contrastive learning, here we are comparing "codes."

During training, each code serves as a fixed target for backpropagation, while the feature's softmax probabilities over the prototypes provide the prediction for cross-entropy. Each update uses the swapped predictions from the current batch.

Equal Allocation Constraint

To achieve online learning, we only use the features within the "current batch" to calculate the codes, rather than using the entire dataset. This way, the model can update the prototypes through continuously receiving different batches of data, allowing these prototypes to gradually adapt to the diversified feature distribution.

However, if there are no constraints on the prototypes, it's easy for all features to be mapped to the same vector, which can cause model collapse. To avoid this, the authors introduce an "equal allocation constraint," requiring that the allocation of prototypes within each batch be as uniform as possible, ensuring that each prototype receives a similar number of samples.

Suppose the current batch contains BB feature vectors, denoted as

Z=[z1,…,zB],Z = [z_1, \dots, z_B],

and KK prototypes are denoted as

C=[c1,…,cK].C = [c_1, \dots, c_K].

We aim to optimize an allocation matrix QQ to achieve the best matching between features and prototypes. The objective function

Tr(Q⊤C⊤Z)\mathrm{Tr}(Q^\top C^\top Z)

represents the total similarity between all features and prototypes, while the entropy regularization term ϵ H(Q)\epsilon\,H(Q) (where H(Q)=−∑ijQijlog⁡QijH(Q) = -\sum_{ij} Q_{ij} \log Q_{ij}) is used to prevent overly concentrated allocations. This design encourages the model to maintain discriminative power while avoiding all features being concentrated on just a few prototypes.

In actual optimization, we restrict QQ to a specific set Q\mathcal{Q}, defined as:

Q={Q∈R+K×B  ∣  Q 1B=1K 1K,Q⊤1K=1B 1B},\mathcal{Q} = \Bigl\{ Q \in \mathbb{R}_{+}^{K \times B} \;\Big|\; Q\,\mathbf{1}_{B} = \tfrac{1}{K}\,\mathbf{1}_{K}, \quad Q^\top \mathbf{1}_{K} = \tfrac{1}{B}\,\mathbf{1}_{B} \Bigr\},

Here 1K\mathbf{1}_K and 1B\mathbf{1}_B are all-one vectors of dimensions KK and BB. In the K×BK\times B matrix, rows represent prototypes and columns represent samples. Each row sums to 1/K1/K and each column to 1/B1/B. In aggregate, each prototype receives the equivalent of B/KB/K samples' assignment mass; these are soft assignments, not necessarily an integer number of hard labels.

The smoothing weight ϵ\epsilon in the entropy term matters: if it is too large, the assignments become too uniform and all samples may converge toward the same representation. The authors use a small ϵ\epsilon in their experiments.

The authors retain continuous soft assignments Q∗Q^* and solve for them with the Sinkhorn-Knopp algorithm. It repeatedly normalizes rows and columns to compute the diagonal matrices Diag(u)\mathrm{Diag}(u) and Diag(v)\mathrm{Diag}(v). At full convergence, the result meets the required row and column sums and has the form:

Q∗=Diag(u) exp⁡(C⊤Zϵ) Diag(v).Q^* = \mathrm{Diag}(u)\,\exp \Bigl(\tfrac{C^\top Z}{\epsilon}\Bigr)\,\mathrm{Diag}(v).

Three iterations give good model performance in the experiments; with finitely many iterations, this is an approximate solution rather than exact satisfaction of the constraints. The computation can also run efficiently on a GPU.

Additionally, in small batch cases, when the number of features BB in a batch is much smaller than the number of prototypes KK, a single batch may not guarantee that every prototype receives enough samples. To address this, the authors keep the features from previous batches, accumulating them into a larger feature set, and then perform the allocation computation collectively. However, during backpropagation, only the samples from the current batch are updated.

For a batch size of 256, the small-batch experiment stores 3,840 features from the preceding 15 batches; the current 256 are included when computing assignments. Under the compared setting, MoCo v2 stores 65,536 features. These figures compare feature queue sizes, not the methods' total memory use.

Multiple Crops

Previous studies mostly used random crops to capture the relationships of different parts of an image (such as scenes or objects). Through cropping different regions, the model can learn both local features and the global structure.

More crops, or views, also increase the computation and memory required to encode them. Comparing every view with every other view would additionally make the number of pairs grow quadratically with the number of views.

To provide more views while limiting the added cost, the authors propose a multi-crop strategy:

  1. Two standard resolution crops: These retain more image detail and provide the features used to compute codes.
  2. Additional VV low-resolution crops: These low-resolution crops only cover part of the image, but since their size is smaller, the computational cost is lower.

Low-resolution crops cost less to process than high-resolution ones, allowing more views of the same image with a smaller added burden than equally many full-resolution crops. The actual time and memory still depend on the crop configuration; the paper's Table 4 shows that the larger multi-crop setting uses more of both.

The original loss function was used for the swapped prediction loss between two views. Under the multiple crops strategy, the loss function is generalized as:

L(zt1,zt2,…,ztV+2)=∑i∈{1,2}∑v=1v≠iV+2ℓ(ztv,qti)L(z_{t_1}, z_{t_2}, \dots, z_{t_{V+2}}) = \sum_{i \in \{1,2\}} \sum_{\substack{v=1 \\ v\neq i}}^{V+2} \ell(z_{t_v}, q_{t_i})

Here, zt1z_{t_1} and zt2z_{t_2} represent the features from the two standard resolution crops. The remaining ztvz_{t_v} (v=3,…,V+2v=3,\dots,V+2) represent the features from the low-resolution crops.

The authors only use the high-resolution crops to calculate the "codes" and then use these codes to compare the features of all other crops (including the low-resolution ones).

Discussion

The authors utilize the features learned by SwAV and perform transfer learning on multiple datasets to validate the generalizability and practicality of these features. They introduce several improvements from SimCLR, including:

  • LARS (Layer-wise Adaptive Rate Scaling): Helps stabilize the learning rate during large-batch training.
  • Cosine Learning Rate: Uses a cosine annealing strategy to adjust the learning rate, making the training process smoother.
  • MLP Projection Head: Projects the features through an additional fully connected layer (MLP) to further enhance feature representation.

Experimental Results

We focus on evaluating the unsupervised features on ImageNet.

The method involves extracting features from a ResNet-50 network (pretrained with SwAV), fixing them, and not fine-tuning. A linear classifier is then trained on these "frozen" features.

The results are shown in the table below:

SwAV reaches 75.3% Top-1 accuracy in linear classification, 4.2 percentage points above the MoCo v2 result of 71.1% shown in the figure and 1.2 points below the fully supervised result of 76.5%. In this experiment, SwAV uses batches of 4096 and trains for 800 epochs.

The right side of the figure shows the performance of ResNet-50 models with different widths on unsupervised feature learning.

As model width increases, SwAV's performance also improves. On the widest model in the figure, the gap to the supervised model of the same width is only 0.6 percentage points. These results describe ImageNet linear evaluation; transfer performance must be checked separately on downstream tasks.

Ablation Study: Multiple Crops

The above figure demonstrates the effect of the multi-crop strategy. The experiment was conducted on the ImageNet dataset, using ResNet-50 as the base network.

The table on the left compares clustering-based methods with contrastive instance methods; the graph on the right shows SwAV results at different training lengths.

The experimental results show:

  • Without multiple crops: SwAV reaches 70.1% and DeepCluster-v2 70.2%, exceeding SimCLR's 68.2% by 1.9 and 2.0 percentage points, respectively.
  • With multiple crops: SwAV reaches 74.1% and DeepCluster-v2 74.3%, exceeding SimCLR's 70.6% by 3.5 and 3.7 points, respectively.

In this comparison, which uses the same augmentations, training length, and batch size, the clustering methods achieve higher Top-1 accuracy than SimCLR. This alone does not show that every clustering method is better than every instance contrastive method.

From the results, SwAV and DeepCluster-v2 have very similar performances, but the key difference lies in the training modes:

  • DeepCluster-v2 needs multiple passes over the dataset to update cluster centers and assignments, making it unsuitable for online training. On a very large dataset where only one pass is practical, it cannot operate in that setting.
  • SwAV swaps assignments at the batch level and updates online, without repeatedly traversing the entire dataset solely to reassign codes.

From another perspective, DeepCluster-v2 can be viewed as a special case of SwAV's swapping mechanism: in DeepCluster-v2, swapping happens between different training epochs, using the previous epoch's assignment results to predict the assignment for a current augmented view; whereas SwAV performs the swapping within the same batch, making it more suited for online training environments.

Conclusion

SwAV uses a set of learnable cluster centers, no longer directly comparing image features. Instead, it maps the features to the cluster centers they represent and then swaps information between different views.

This process acts as a kind of intermediary language, helping the model maintain consistency in unsupervised learning and capture the internal structure and relationships of images. Its ImageNet linear-evaluation result approaches supervised training, and the paper also reports stronger results than supervised pretraining on the evaluated transfer tasks. Together, these show the potential of this design for unsupervised learning.