Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Publication
- NeurIPS 2020Conference
- First submitted to arXiv
- arXiv:2006.09882
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{contrastivelearningswav2020,
title = {{Unsupervised Learning of Visual Features by Contrasting Cluster Assignments}},
author = {Mathilde Caron and Ishan Misra and Julien Mairal and Priya Goyal and Piotr Bojanowski and Armand Joulin},
year = {2020},
eprint = {2006.09882},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2006.09882}
}
There has been some debate between the MoCo and SimCLR series for a while.
Previously, we focused on comparing positive and negative samples in pairs. This time, let's explore a different approach: how about trying clustering?
Problem Definition
The structure of contrastive learning has largely been established, with several main ingredients:
- Using contrastive loss: InfoNCE-style losses, including SimCLR’s NT-Xent, is one commonly used loss function.
- Applying image transformations: The features of the same image are pulled together, while features of different images are pushed apart.
- A large number of negative samples: More negatives can help, depending on the data and training setup.
Of course, there are many variations in the details, such as using momentum encoders or memory banks.
The authors of this paper believe: "The large computational resources are still a significant issue!"
What if we change the approach and stop focusing solely on contrastive loss, instead turning it into a "prediction" problem?
We have the model "predict" the cluster assignments of another view, rather than directly comparing the features. The advantage here is that we no longer need to compute all pairwise comparisons but can achieve the same effect through cluster assignments.
Solving the Problem
Model Architecture
Looking at this diagram, you’ve essentially understood the paper.
First, the left side shows the currently widely accepted contrastive learning architecture:
In the diagram, you can see that an image undergoes various transformations (transformations ) to produce two augmented views (e.g., ).
These two augmented views are each passed through a convolutional network (or any feature extraction network) to output feature vectors ().
In contrastive instance learning, the method "directly compares these two feature vectors", such as calculating their similarity (similar to methods like SimCLR, MoCo, etc.). The model aims to bring features from the same image closer together in feature space, while features from different images should be pushed farther apart.
The right side shows the method proposed in this paper, let’s zoom in:
In SwAV, the authors first assign each feature (e.g., ) to a set of trainable "prototype vectors" (hereafter referred to as prototypes), obtaining corresponding "codes" .
This step is represented by "Prototypes C" and "Codes" in the diagram.
Next, the core of SwAV is the "Swapped Prediction", meaning:
- Use the code corresponding to the feature of view 1 to guide what the feature of view 2 should learn.
- Use the code corresponding to the feature of view 2 to guide what the feature of view 1 should learn.
Thus, SwAV does not directly compare and . Instead, the features of one view predict the other view's assignment to the prototypes, encouraging consistent assignments across views of the same image.
Since prototypes are a set of learnable vectors, they can be seen as "cluster centers" used to map feature vectors to "codes," forming a clustering effect. During training, the model simultaneously learns these prototypes, placing them in positions in feature space that better distinguish different images.
Online Clustering
An important feature of SwAV is "Online Clustering."
Let’s take a look at how the authors calculate the "code" for each image online and update the prototypes:
For each image , a transformation is randomly sampled from a set of transformations (e.g., random cropping, horizontal flipping, color jittering), generating an augmented image . Then, a convolutional neural network is used to extract feature vectors, and the feature vectors are projected onto the unit sphere, obtaining , i.e., normalizing the length of the output vector to 1.
At this point, there is not much difference from previous contrastive learning papers.
Finally, is mapped to trainable "prototypes" , resulting in a "code" .
After obtaining the cluster code for each image, the next step is to perform the swapped prediction:
Two augmented views correspond to the same image. Let represent the feature and code for view , and represent the feature and code for view .
- The authors want to use to "predict" , and to "predict" .
- This differs from directly comparing feature vectors in contrastive learning, here we are comparing "codes."
During training, each code serves as a fixed target for backpropagation, while the feature's softmax probabilities over the prototypes provide the prediction for cross-entropy. Each update uses the swapped predictions from the current batch.
Equal Allocation Constraint
To achieve online learning, we only use the features within the "current batch" to calculate the codes, rather than using the entire dataset. This way, the model can update the prototypes through continuously receiving different batches of data, allowing these prototypes to gradually adapt to the diversified feature distribution.
However, if there are no constraints on the prototypes, it's easy for all features to be mapped to the same vector, which can cause model collapse. To avoid this, the authors introduce an "equal allocation constraint," requiring that the allocation of prototypes within each batch be as uniform as possible, ensuring that each prototype receives a similar number of samples.
Suppose the current batch contains feature vectors, denoted as
and prototypes are denoted as
We aim to optimize an allocation matrix to achieve the best matching between features and prototypes. The objective function
represents the total similarity between all features and prototypes, while the entropy regularization term (where ) is used to prevent overly concentrated allocations. This design encourages the model to maintain discriminative power while avoiding all features being concentrated on just a few prototypes.
In actual optimization, we restrict to a specific set , defined as:
Here and are all-one vectors of dimensions and . In the matrix, rows represent prototypes and columns represent samples. Each row sums to and each column to . In aggregate, each prototype receives the equivalent of samples' assignment mass; these are soft assignments, not necessarily an integer number of hard labels.
The smoothing weight in the entropy term matters: if it is too large, the assignments become too uniform and all samples may converge toward the same representation. The authors use a small in their experiments.
The authors retain continuous soft assignments and solve for them with the Sinkhorn-Knopp algorithm. It repeatedly normalizes rows and columns to compute the diagonal matrices and . At full convergence, the result meets the required row and column sums and has the form:
Three iterations give good model performance in the experiments; with finitely many iterations, this is an approximate solution rather than exact satisfaction of the constraints. The computation can also run efficiently on a GPU.
Additionally, in small batch cases, when the number of features in a batch is much smaller than the number of prototypes , a single batch may not guarantee that every prototype receives enough samples. To address this, the authors keep the features from previous batches, accumulating them into a larger feature set, and then perform the allocation computation collectively. However, during backpropagation, only the samples from the current batch are updated.
For a batch size of 256, the small-batch experiment stores 3,840 features from the preceding 15 batches; the current 256 are included when computing assignments. Under the compared setting, MoCo v2 stores 65,536 features. These figures compare feature queue sizes, not the methods' total memory use.
Multiple Crops
Previous studies mostly used random crops to capture the relationships of different parts of an image (such as scenes or objects). Through cropping different regions, the model can learn both local features and the global structure.
More crops, or views, also increase the computation and memory required to encode them. Comparing every view with every other view would additionally make the number of pairs grow quadratically with the number of views.
To provide more views while limiting the added cost, the authors propose a multi-crop strategy:
- Two standard resolution crops: These retain more image detail and provide the features used to compute codes.
- Additional low-resolution crops: These low-resolution crops only cover part of the image, but since their size is smaller, the computational cost is lower.
Low-resolution crops cost less to process than high-resolution ones, allowing more views of the same image with a smaller added burden than equally many full-resolution crops. The actual time and memory still depend on the crop configuration; the paper's Table 4 shows that the larger multi-crop setting uses more of both.
The original loss function was used for the swapped prediction loss between two views. Under the multiple crops strategy, the loss function is generalized as:
Here, and represent the features from the two standard resolution crops. The remaining () represent the features from the low-resolution crops.
The authors only use the high-resolution crops to calculate the "codes" and then use these codes to compare the features of all other crops (including the low-resolution ones).
Discussion
The authors utilize the features learned by SwAV and perform transfer learning on multiple datasets to validate the generalizability and practicality of these features. They introduce several improvements from SimCLR, including:
- LARS (Layer-wise Adaptive Rate Scaling): Helps stabilize the learning rate during large-batch training.
- Cosine Learning Rate: Uses a cosine annealing strategy to adjust the learning rate, making the training process smoother.
- MLP Projection Head: Projects the features through an additional fully connected layer (MLP) to further enhance feature representation.
Experimental Results
We focus on evaluating the unsupervised features on ImageNet.
The method involves extracting features from a ResNet-50 network (pretrained with SwAV), fixing them, and not fine-tuning. A linear classifier is then trained on these "frozen" features.
The results are shown in the table below:
SwAV reaches 75.3% Top-1 accuracy in linear classification, 4.2 percentage points above the MoCo v2 result of 71.1% shown in the figure and 1.2 points below the fully supervised result of 76.5%. In this experiment, SwAV uses batches of 4096 and trains for 800 epochs.
The right side of the figure shows the performance of ResNet-50 models with different widths on unsupervised feature learning.
As model width increases, SwAV's performance also improves. On the widest model in the figure, the gap to the supervised model of the same width is only 0.6 percentage points. These results describe ImageNet linear evaluation; transfer performance must be checked separately on downstream tasks.
Ablation Study: Multiple Crops
The above figure demonstrates the effect of the multi-crop strategy. The experiment was conducted on the ImageNet dataset, using ResNet-50 as the base network.
The table on the left compares clustering-based methods with contrastive instance methods; the graph on the right shows SwAV results at different training lengths.
The experimental results show:
- Without multiple crops: SwAV reaches 70.1% and DeepCluster-v2 70.2%, exceeding SimCLR's 68.2% by 1.9 and 2.0 percentage points, respectively.
- With multiple crops: SwAV reaches 74.1% and DeepCluster-v2 74.3%, exceeding SimCLR's 70.6% by 3.5 and 3.7 points, respectively.
In this comparison, which uses the same augmentations, training length, and batch size, the clustering methods achieve higher Top-1 accuracy than SimCLR. This alone does not show that every clustering method is better than every instance contrastive method.
From the results, SwAV and DeepCluster-v2 have very similar performances, but the key difference lies in the training modes:
- DeepCluster-v2 needs multiple passes over the dataset to update cluster centers and assignments, making it unsuitable for online training. On a very large dataset where only one pass is practical, it cannot operate in that setting.
- SwAV swaps assignments at the batch level and updates online, without repeatedly traversing the entire dataset solely to reassign codes.
From another perspective, DeepCluster-v2 can be viewed as a special case of SwAV's swapping mechanism: in DeepCluster-v2, swapping happens between different training epochs, using the previous epoch's assignment results to predict the assignment for a current augmented view; whereas SwAV performs the swapping within the same batch, making it more suited for online training environments.
Conclusion
SwAV uses a set of learnable cluster centers, no longer directly comparing image features. Instead, it maps the features to the cluster centers they represent and then swaps information between different views.
This process acts as a kind of intermediary language, helping the model maintain consistency in unsupervised learning and capture the internal structure and relationships of images. Its ImageNet linear-evaluation result approaches supervised training, and the paper also reports stronger results than supervised pretraining on the evaluated transfer tasks. Together, these show the potential of this design for unsupervised learning.