Back to paper library
arXiv 2021ICCV 2021Contrastive learningPaper notes

MoCo v3

ViT Expedition Team

Original paper

An Empirical Study of Training Self-Supervised Vision Transformers

Paper authors

Xinlei Chen · Saining Xie · Kaiming He

Publication
ICCV 2021Conference
First submitted to arXiv
arXiv:2104.02057
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{contrastivelearningmocov32021,
  title = {{An Empirical Study of Training Self-Supervised Vision Transformers}},
  author = {Xinlei Chen and Saining Xie and Kaiming He},
  year = {2021},
  eprint = {2104.02057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.02057}
}
On this page

After five years of representation-learning papers, it felt as though the dust had more or less settled.

At this time, however, a major update emerged in the computer vision field: ViT was released (October 2020).

The MoCo research team couldn’t contain their excitement and eagerly began exploring this brand-new architecture.

Defining the Problem

After the ViT architecture was released, the authors of the original paper indicated that when using a BERT-like MASK mechanism for pre-training on this architecture, the results were somewhat disappointing.

The original ViT paper’s masked-patch pretraining did not match its strongest supervised results under the tested setup. That experiment does not show that image masking or self-supervised learning is inherently ineffective; data, targets, and training recipes also matter.

This conclusion in the ViT paper indeed sparked a lot of researchers to try their hand at it:

Does unsupervised learning really not work? Or was it just your approach that was wrong?

Thus, the authors of the MoCo paper also decided to join in on the exploration of the ViT architecture to see if introducing a twin network structure for representation learning could yield good results in pre-training the model.

Solving the Problem

This paper does not introduce a new method or architecture. The authors build a ViT baseline from MoCo, adding a predictor and symmetric loss and using in-batch negatives instead of the queue from MoCo v1/v2.

The specific process can be directly seen in the algorithmic structure description provided in the paper. In fact, once this section is understood, the paper is essentially covered:

Let's go through the diagram:

  1. Prepare two encoders:

    • The first encoder’s architecture is: backbone + proj mlp + pred mlp
    • The second encoder’s architecture is: backbone + proj mlp (momentum update)
  2. Extract a batch xx (containing NN images) from the data, and apply data augmentation to generate x1,x2x_1, x_2 for each image. Negatives come from the same batch, without the older cross-batch MoCo queue.

  3. With L2-normalized outputs, use fqf_q to get query vectors q1,q2q_1, q_2 for x1,x2x_1, x_2 respectively; use fkf_k to obtain key vectors k1,k2k_1, k_2.

  4. Call the function ctr(q,k)ctr(q, k) to compute the contrastive loss:

    • The indices for the positive sample pair are on the diagonal (using labels = range(N)).
    • After dividing logits by τ\tau, use the cross-entropy loss function to compute the loss.
    • Finally, the loss is multiplied by 2τ2\tau as the output.
    • The final total loss is ctr(q1,k2)+ctr(q2,k1)ctr(q_1, k_2) + ctr(q_2, k_1).

The loss function here is the InfoNCE loss, which was previously mentioned, and its formula is:

Lq=−log⁡exp⁡(q⋅k+τ)exp⁡(q⋅k+τ)+∑k−exp⁡(q⋅k−τ).L_q = -\log \frac{\exp\left(\frac{q\cdot k^+}{\tau}\right)}{\exp\left(\frac{q\cdot k^+}{\tau}\right)+\sum_{k^-}\exp\left(\frac{q\cdot k^-}{\tau}\right)}.

Where:

  • qq is the query vector;
  • k+k^+ is the positive sample vector, obtained from the same image as qq but using another encoder fkf_k;
  • {k−}\{k^-\} is the negative sample set, consisting of key vectors from other images in the same batch;
  • τ\tau is the temperature hyperparameter, which controls the smoothness of the distribution.

The objective of this loss is to bring qq and k+k^+ closer while pushing qq away from all k−k^-.

After backpropagation, the optimizer updates fqf_q, and then fkf_k is updated using momentum to ensure the key encoder remains stable.


So far, these are all familiar ingredients. The paper's focus is therefore on the experiments; let's look at them carefully.

Discussion

In this paper, the authors monitor the kNN curve to observe the changes during training.

The kNN curve is a tool used to monitor the quality of the feature representations learned by the model in real-time. The process works as follows:

  1. Feature extraction: During training, the model extracts features from input images. The paper uses class-token features to monitor the representation without training another classifier.

  2. kNN classification: For each evaluation sample, k-nearest neighbors (kNN) are found in a small feature memory bank, and their labels are combined to make a prediction. The bank covers about 10% of ImageNet and is updated each iteration with training-sample features; the paper follows an existing kNN implementation rather than specifying a simple majority vote here.

  3. Curve generation: The authors evaluate kNN accuracy densely, for example every tens of iterations, and plot it over training. The curve tracks representation quality at different stages, exposing dips that a once-per-epoch check might miss.

  4. Revealing training stability: By observing the kNN curve, researchers can discover that even when the loss function seems to decrease and the final accuracy appears reasonable, the quality of the model’s feature representation (such as kNN classification accuracy) may still experience "plummeting" or fluctuations. These fluctuations suggest instability during training, which might not cause divergence but could result in hidden degradation of final performance.

In summary, the kNN curve is an evaluation method based on k-nearest neighbor classification, used to track the stability and quality of the feature representations learned by the model during training, thus revealing potential instability issues.

ViT Instability

In theory, replacing ResNet with ViT as the backbone in a twin network self-supervised framework isn't difficult; however, in practice, training ViT faces significant instability issues. This instability is not immediately apparent from the final accuracy, as even when the training results seem "normal" on the surface, the process may hide local failures or "partial successes."

  • Impact of Batch Size

    ViT models are computationally heavy, and large batch training provides more negative samples, which aids contrastive learning. However, when the batch size is too large (e.g., 4k or 6k), significant "fluctuations" or "drops" appear in the training curve.

    As shown in the figure above:

    • Batch size 1k: Linear probing accuracy is about 71.5%, with a smooth curve.
    • Batch size 2k: Accuracy rises to about 72.6%, maintaining smoothness.
    • Batch size 4k: The curve shows noticeable "dips," with accuracy slightly dropping to 72.2%.
    • Batch size 6k: More evident fluctuations, with a large drop, and accuracy decreases to 69.7%.

    Larger batches offer more in-batch negatives, yet the 4096 and 6144 settings in this figure show kNN dips. The authors hypothesize a kind of "partial restart" in the optimization trajectory; they do not directly observe parameters restarting. Final linear accuracy can differ by only about 1–3 percentage points, so a single endpoint number can hide the issue.

  • Impact of Learning Rate

    To handle large batch training, a linear learning rate adjustment rule is often used, which is given by the formula:

    Code
    $$
    \text{Effective LR} = \text{base lr} \times \frac{\text{BatchSize}}{256}.
    $$
    

    Experimental results show:

    • When using a smaller learning rate (e.g., lr = 0.5e-4), training is more stable but may suffer from underfitting, leading to lower accuracy (70.4%).
    • A moderate learning rate (e.g., lr = 1.0e-4) achieves better fitting, with accuracy reaching 72.2%.
    • At lr = 1.5e-4, more dips appear and final linear accuracy falls to 71.7%.

    The learning rate must balance stability and adequate fitting. Too low a rate can underfit; too high a rate brings more instability and worse final accuracy in this experiment.

  • Impact of Optimizer

    Some recent self-supervised methods have adopted the LARS optimizer, which is designed for large batch training, while LAMB can be viewed as the LARS version of AdamW.

    The authors tested using the LAMB optimizer to see the results:

    • At lr = 5e-4, LAMB reaches 72.5%, slightly above AdamW in this comparison.
    • Relative to that 72.5%, lr = 6e-4 and 8e-4 fall to 70.9% and 66.5%, lower by 1.6 and 6.0 percentage points.

    The authors point out that although LAMB's gradient changes are smoother, this smoothness might cause unstable gradient issues to accumulate gradually, ultimately affecting model performance. LAMB is highly sensitive to learning rate, and its effect varies under different settings, making it difficult to compare different architecture designs without conducting additional learning rate searches.

    For this reason, the authors chose to use the more stable AdamW optimizer in other experiments.

Solving ViT Instability Issues

During training, the authors observed that sudden changes in gradients caused noticeable "dips" in the training curve, as shown in the figure below:

Comparing the first and last layers in the figure, the gradient spikes appear earlier in the patch projection layer and tens of iterations later in the last layer.

This phenomenon suggests that instability originates in the shallow layers first, potentially affecting the performance of deeper layers. Based on this observation, the authors hypothesize that instability primarily starts in the first layer.

To prevent the gradient fluctuations caused by this layer, the authors propose freezing the patch projection layer during training. The practical implementation of this technique is to use the stop-gradient operation, which fixes the weights of this layer and instead uses a fixed random projection to embed the image patches.

The experimental results are shown below:

Compared to a learnable patch projection, using a random (frozen) patch projection results in smoother training curves and improved stability.

With a learning rate of lr = 1.5e-4, the improved stability increases the final linear probing accuracy by 1.7 percentage points, reaching 73.4%. As the learning rate increases, this stability improvement becomes more evident (increases by 0.4, 0.6, and 1.7 percentage points).

In addition to fixing the random weights, the authors also tried using BatchNorm or WeightNorm on the patch projection layer but found no improvement in stability, with results similar to the original configuration. When gradient clipping is applied to this layer with a sufficiently small threshold, it can have a similar effect to freezing the weights.

In this paper's ViT-B/16, a 16×16 RGB patch has 768 input dimensions and is projected to 768 output dimensions, so the projection does not reduce dimensionality. The authors argue that a fixed random projection can therefore retain enough information here. The experiment shows that learning this first layer is not necessary for these results; narrowing the learnable parameter space also points to an optimization issue, without proving that all instability starts in that layer.

While this technique alleviates instability, when the learning rate is too high, the entire model can still become unstable. Therefore, this is only a partial solution and does not fundamentally eliminate the problem.

Comparison of Self-Supervised Learning Frameworks

The authors tested on four frameworks: MoCo v3, SimCLR, BYOL, and SwAV, ensuring fairness by using the same random projection technique in all experiments, and performing a parameter search for learning rate (lr) and weight decay (wd).

MoCo v3 reaches 72.5% on ViT-S/16 and 76.5% on ViT-B/16, the highest values in the two ViT columns. The R50 row uses 800 epochs while the ViT rows use 300, so the plotted relative positions describe these settings and cannot be attributed entirely to the backbone.

Ablation Study

  • Position Embedding:

    Three options were compared here: sin-cos (default), learnable, and no position embedding.

    Sin-cos gives 76.5% linear accuracy, learned embeddings 76.1%, and no positional embedding 74.9%. In this setting the model still learns a representation useful for linear classification without explicit position embeddings, though it scores 1.6 percentage points below the default.

  • The Role of Class Token:

    The ablation study compared using [CLS] versus not using [CLS].

    When [CLS] was removed but the final LayerNorm (LN) was retained for pooling, performance significantly dropped to 69.7%. However, directly removing LN and using global average pooling yielded a result close to the original (76.3%).

    The results show that [CLS] is not absolutely necessary, but the use of LN has a significant impact on the final performance.

  • BatchNorm (BN) in the MLP Head:

    The ViT model itself does not contain BN, using it only in the MLP head.

    With BN, accuracy is 76.5%. Without it, the authors change the batch size to 2048 to obtain convergence, and accuracy is 74.4%, a 2.1-percentage-point difference. This comparison changes both BN and batch size.

  • Prediction Head:

    MoCo v3 by default includes a prediction MLP head. In the ablation experiment, when the prediction head was removed, the accuracy dropped from 76.5% to 75.5%, indicating that the prediction head can further improve performance, but is not essential.

  • Momentum Encoder:

    Different momentum coefficients mm were compared (0, 0.9, 0.99, 0.999).

    At the default m=0.99m = 0.99, accuracy reaches 76.5%; at m=0m = 0, it is 74.3%, a 2.2-percentage-point gap. The m=0m=0 variant still has a predictor and stop-gradient key branch, so it is only similar to SimCLR in some respects.

  • Impact of Training Length:

    The results of training for 300 epochs versus 600 epochs were compared.

    The smaller ViT-S/16 model improved from 72.5% at 300 epochs to 73.4% at 600 epochs; however, the ViT-B/16 model only improved from 76.5% to 76.7%. This shows that smaller models benefit more from extended training, while larger models’ gains tend to saturate.

Comparison with Other Methods

The table above shows a comparison of MoCo v3 with iGPT and masked patch prediction methods.

The table mixes evaluation settings. MoCo v3 improves on iGPT in the ImageNet linear-probing comparison with a smaller model. The original ViT masked-patch result instead uses JFT-300M pretraining and end-to-end ImageNet fine-tuning, so it is not the same-data linear-probing comparison. Separately, the reported ViT-L self-supervised result of 77.6% exceeds the cited supervised-from-scratch result of 76.53%.

The following chart shows a trade-off comparison between parameter count and linear probing accuracy to compare large-scale ResNet models with self-supervised ViT models:

The horizontal axis typically represents the model’s parameter count (i.e., model size), while the vertical axis represents the linear probing accuracy. This display helps to visualize how model performance changes as the model size (parameter count) increases. It provides an intuitive understanding of the balance between resource consumption (parameter count) and final performance (accuracy) and the effectiveness of self-supervised learning methods across different model sizes.

In the graph, baseline MoCo v3 ViTs sit slightly above ordinary ResNet SimCLR v2 at smaller model sizes and are close at larger sizes; some SK-ResNet SimCLR v2 and wide BYOL points sit higher. The BN and /7-patch ViT variants are plotted separately from baseline ViT. The different methods' points are not controlled ablations at identical parameter counts and training settings.

Conclusion

Well, after reading the whole paper, it still does not introduce a new method. So why call it MoCo v3?

The familiar recipe behaves differently on ViT, and MoCo v3 leads the compared ViT results in Table 4. The stability study gives this version its own story.

So, giving it another version number is fair enough, right?