Improved Baselines with Momentum Contrastive Learning
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2003.04297
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{contrastivelearningmocov22020,
title = {{Improved Baselines with Momentum Contrastive Learning}},
author = {Xinlei Chen and Haoqi Fan and Ross Girshick and Kaiming He},
year = {2020},
eprint = {2003.04297},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2003.04297}
}
FAIR introduced MoCo v1, but before they could even enjoy their success, Google presented SimCLR, which showed a significant performance boost!
That landed quite a punch on FAIR. Surely they could not leave it there! About a month after SimCLR, FAIR took some of its good ideas and released MoCo v2. At least, that is how I picture this research exchange.
The message was clear: MoCo could still be the better choice!
Problem Definition
SimCLR made three major improvements in its implementation:
- Very large batch sizes (4k or 8k), ensuring a sufficient number of negative samples.
- MLP head: The output features of the encoder are passed through a 2-layer MLP (with ReLU) before applying contrastive loss. After training, the encoder’s output (without the MLP output) is usually used for downstream tasks.
- Stronger data augmentation: This includes stronger color jittering, blurring, etc.
However, despite the excellent performance, the batch size in SimCLR is really too large, and most people cannot run it.
Thus, the authors in this paper attempt to integrate SimCLR’s two major improvements into MoCo:
- MLP projection head
- Stronger data augmentation
They found that these improvements are also effective within the MoCo framework and can achieve results surpassing SimCLR even in a typical 8-GPU environment (without requiring ultra-large batch sizes).
This is clearly more user-friendly for general users and makes contrastive learning research easier to reproduce.
Solution
The diagram above shows the architectural differences between SimCLR and MoCo.
SimCLR is an end-to-end mechanism, requiring a batch with enough negative samples, typically requiring a batch size of 4k to 8k.
In contrast, MoCo uses a momentum encoder to generate relatively consistent key representations. The negative keys are stored in a queue shared across batches. Even with a small batch, that queue can provide plenty of negatives.
Thus, MoCo does not need large batches to retain many negative samples and requires fewer hardware resources.
Enhanced MLP Head
The authors replaced the original fully connected (fc) projection head in MoCo with a 2-layer MLP (2048 units in the intermediate layer, with ReLU), testing different temperature parameters .
The authors report the temperature sweep in the text, separately from the table above. At , replacing fc with MLP raises ImageNet linear accuracy from 60.6% to 62.9%. With the MLP and , it reaches 66.2%. Table 1 compares this tuned setting with the baseline; it does not show the temperature sweep.
However, in transfer learning for object detection, the improvement from the MLP head is smaller. This raises a common question:
- "Is a higher linear classification score always better? Does it directly correlate with actual downstream tasks?"
Sometimes, this is not entirely true. Linear classification scores are a reference indicator; evaluate each downstream task separately.
Stronger Data Augmentation
The augmentation change highlighted here is Gaussian blur from SimCLR. The authors also tested stronger color distortion but reported limited additional benefit. Without the MLP, the revised augmentation raises ImageNet linear accuracy from 60.6% to 63.4%; the detection gain is comparable to or greater than that of adding the MLP.
With both MLP and stronger augmentation, ImageNet linear accuracy reaches 67.3%, and detection improves too. Interestingly, blur augmentation alone gives better VOC detection AP than the MLP alone, although its ImageNet linear score is lower. The later 67.5% result also adds a cosine learning-rate schedule.
Discussion
Comparison with SimCLR
In the longer-run comparison here, SimCLR uses batch size 4096 for 1000 epochs and reaches 69.3% ImageNet linear accuracy.
The authors integrated the MLP head, stronger augmentation and cosine learning-rate schedule into MoCo under a more feasible hardware setup (8-GPU, batch size = 256), creating "MoCo v2."
The results show:
- At 200 epochs and batch size 256, MoCo v2 reaches 67.5%, 5.6 percentage points above SimCLR’s 61.9% under those same two conditions. It also exceeds SimCLR’s 66.6% at 200 epochs with batch size 8192.
- At 800 epochs and batch size 256, MoCo v2 reaches 71.1%, above SimCLR’s 69.3% at 1000 epochs and batch size 4096. Those two longer runs differ in both duration and batch size, so they are not a matched-setting comparison.
In other words, MoCo can use the queue to handle plenty of negative samples without an enormous batch. Add the MLP head, stronger augmentation and cosine learning-rate schedule, and MoCo v2 wins these reported comparisons.
Computational Load and Resource Consumption
The authors also compared the memory and time costs of the end-to-end mechanism and the MoCo mechanism on eight V100 GPUs with 16 GB each.
At batch size 4096, the authors estimate that end-to-end training would require 93 GB per GPU, beyond the 16 GB capacity used here; they report no measured training time for that row. Even at the shared batch size of 256, end-to-end requires 7.4 GB per GPU and about 65 hours for 200 epochs, versus MoCo’s 5.0 GB and 53 hours, because it backpropagates through both query and key encoders.
MoCo backpropagates only through the query branch. Its key encoder is still updated by the momentum rule, so saying that the key encoder never changes would be incorrect.
Conclusion
In short, the two SimCLR improvements tested here also work in MoCo. Thanks to MoCo's queue, there is no need for a huge GPU batch to get these results. Under the paper's training setup, that is a pretty economical way to get strong performance.
In this round, FAIR successfully made a comeback against Google.