Bootstrap your own latent: A new approach to self-supervised Learning
- Publication
- NeurIPS 2020Conference
- First submitted to arXiv
- arXiv:2006.07733
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{contrastivelearningbyol2020,
title = {{Bootstrap your own latent: A new approach to self-supervised Learning}},
author = {Jean-Bastien Grill and Florian Strub and Florent Altché and Corentin Tallec and Pierre H. Richemond and Elena Buchatskaya and Carl Doersch and Bernardo Avila Pires and Zhaohan Daniel Guo and Mohammad Gheshlaghi Azar and Bilal Piot and Koray Kavukcuoglu and Rémi Munos and Michal Valko},
year = {2020},
eprint = {2006.07733},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2006.07733}
}
While everyone is actively discussing how to design negative samples, the authors of this paper stand out by proposing an approach that doesn’t require negative samples.
As the authors put it: by learning from itself, a model can push past its own limits.
Problem Definition
The core concept of contrastive learning is to pull together representations of the same image from different views (referred to as "positive sample pairs") while pushing apart representations from different images (referred to as "negative sample pairs") to learn representations.
However, these methods require carefully designed negative sample handling strategies, such as large-batch training (SimCLR) or memory banks (MoCo), to ensure effective learning.
Furthermore, these methods heavily rely on image augmentations, with techniques like random cropping proven to be very effective in experiments.
Given how troublesome the design of negative samples can be, is there a way to avoid them altogether?
Solution
Model Architecture
The authors propose the BYOL architecture, which consists of two parts: an online network and a target network:
-
Online Network: The upper half of the figure above, consisting of three parts:
- Encoder : Converts the input image into features.
- Projector : Maps the features into a high-dimensional space, similar to contrastive learning methods like SimCLR.
- Predictor : Learns a nonlinear mapping in the projected space to predict the output of the target network.
-
Target Network:
The target contains an encoder and projector, but no predictor. Its parameters are an exponential moving average (EMA) of the corresponding online parameters . The update rule is as follows:
Here, is the decay factor that controls the update speed of the target network.
Training Process
The BYOL training process begins with a given image dataset . A random image is selected, and two different image augmentations and are applied to generate two different views:
Where , .
For the first augmented view , the online network computes:
For the second augmented view , the target network computes:
At this point, both branches have produced projected representations, but BYOL has no negative examples. If we merely pull the two representations together, every image might end up with the same trivial representation. The authors add a predictor to the online branch to predict the target projection:
Then, and are compared and pulled together. In other words, the framework becomes a "prediction" problem rather than a "contrast" problem. The target branch is stop-gradient in this loss; only the online branch is updated by backpropagation.
Finally, to ensure numerical stability, normalization is applied when comparing these vectors (making their norms equal to 1):
Loss Function
Here the authors do not use InfoNCE. Instead, the loss is the squared distance between the two normalized vectors:
Expanding the computation:
Where is the inner product of two unit vectors, representing the cosine similarity between them.
Additionally, to symmetrize the learning, the views are swapped during training: passes through the online network, and passes through the target network. The same loss is then computed again:
The final total BYOL loss is:
Implementation Details
BYOL uses the same set of image augmentations as SimCLR, with view-dependent application probabilities:
- Random cropping and resizing to resolution.
- Random horizontal flipping.
- Color jittering (randomly changing brightness, contrast, saturation, and hue).
- Grayscale conversion (optional).
- Gaussian blur.
- Solarization.
For the network architecture, BYOL uses ResNet-50 as the base encoder and , and tests deeper ResNet variants (50, 101, 152, 200 layers) and wider variants (1× to 4×) in different experiments.
The detailed network structure is as follows:
- Output feature dimension: 2048 (when width multiplier is 1×).
- Projection layer (MLP):
- First layer: Linear layer with output dimension 4096.
- Batch normalization.
- ReLU activation.
- Final linear layer: Output dimension 256.
- Predictor: Same structure as the projection layer.
Unlike SimCLR, BYOL does not apply batch normalization to the projector output. The paper does not attribute this choice to whether negative examples are used.
Finally, BYOL uses the LARS optimizer with a cosine decay learning rate schedule:
- Train for 1000 epochs, with the first 10 epochs performing learning rate warmup.
- The base learning rate is:
- Weight decay: .
- Exponential moving average parameter : Starting value of 0.996, gradually increasing to 1 over training: Where is the current step, and is the maximum number of steps.
Discussion
Why Doesn’t BYOL Collapse?
In contrastive learning, the design of negative sample pairs ensures that the model does not collapse into a trivial constant representation.
Since BYOL does not have an explicit regularization term to avoid collapse during training, why doesn’t it collapse?
The authors offer a detailed explanation:
-
The target network parameters in BYOL are not updated through gradient descent. Therefore, the update direction for these parameters is not along the gradient of the loss function :
The authors compare this to the learning dynamics of generative adversarial networks (GANs): the two sets of parameters do not jointly descend the gradient of the same loss. Thus, finding that a constant representation minimizes the stated loss does not, by itself, show that BYOL will converge there. This is the authors’ explanation, not a complete proof against collapse.
-
For further analysis, the authors temporarily set aside normalization and the symmetric loss, and assume that the predictor is optimal (i.e., it minimizes the expected squared error):
The optimal should satisfy:
Under these conditions, it is possible to derive the update direction of BYOL’s parameters. The gradient update with respect to is related to the conditional variance:
where represents the -th feature component of .
Under these assumptions, the update direction can be written as the gradient of the expected conditional variance. For square-integrable variables, adding conditioning information cannot increase the expected conditional variance. The expectation matters here: the pointwise inequality printed in the paper does not hold in general.
If we let:
- (target projection)
- (current online projection)
- be additional variability in the training dynamics
Then, we get:
Under the optimal-predictor assumption, discarding information from the online representation alone cannot lower the expected conditional variance. The authors therefore hypothesize that a constant representation would not be a stable outcome of actual training; the argument does not rule out every degenerate case.
-
Finally, let’s analyze the update mechanism for the target network parameters .
Suppose we directly transfer the online network parameters to the target network:
This would indeed quickly transfer the new variability of the online network to the target network, but it could destroy the assumption of optimality for the predictor, leading to instability in training. Therefore, BYOL uses exponential moving average to update the target network:
This smooth update changes the target more slowly. The authors hypothesize that it helps the predictor stay near optimal, an explanation supported by the experiments.
Performance on ImageNet
Following the standard linear evaluation protocol, the authors train a linear classifier on the frozen representations from BYOL pretraining and measure the Top-1 and Top-5 accuracy.
On the ResNet-50 (1×) architecture, BYOL achieves a Top-1 accuracy of 74.3% (Top-5 accuracy of 91.6%), outperforming previous state-of-the-art self-supervised methods by 1.3 percentage points (Top-1) and 0.5 points (Top-5).
BYOL reduces the gap to the supervised learning baseline (76.5%) but is still below the stronger supervised learning baseline (78.9%).
On deeper and wider ResNets, BYOL continues to outperform the other self-supervised methods in the table. ResNet-50 (4×) reaches 78.6%; the best result, 79.6%, uses ResNet-200 (2×) and approaches the corresponding supervised result.
BYOL achieves performance comparable to, or even better than, contrastive learning methods without requiring negative samples and approaches the supervised learning benchmark. Detailed table data is shown below:
Ablation Studies - Part 1
The authors performed a series of ablation studies to explore the impact of key factors in the BYOL design on the final model performance:
-
Impact of Batch Size on Performance:
In contrastive learning methods, the batch size directly affects the number of negative samples. When the batch size is reduced, the number of negative samples decreases, leading to a worse training performance. Since BYOL does not rely on negative samples, it should theoretically be more robust.
The authors compared BYOL with their own reproduction of SimCLR at batch sizes from 128 to 4096 over 300 epochs, measuring the drop under linear evaluation. BYOL changes relatively little from 256 to 4096, with a clearer drop at 128 that the authors associate with batch normalization. SimCLR falls further as the batch shrinks, which the authors associate with fewer available negatives.
-
Impact of Image Augmentation:
Contrastive learning methods are highly dependent on image augmentations, especially color jittering. This is because randomly cropped views often share color information, but the color variation between views from different images is larger. Without color augmentation, contrastive learning might only learn color histogram-based distinctions, rather than deeper features.
The authors compared BYOL with their reproduction of SimCLR as they progressively removed augmentations. With cropping alone, BYOL still drops about 13.1 percentage points from its baseline, to 59.4%, while SimCLR drops about 27.6 points, to 40.3%. This figure supports the narrower conclusion that BYOL is less sensitive under these settings.
-
Impact of Bootstrapping:
One of the core mechanisms of BYOL is using the target network to provide learning targets, where the target network weights are the exponential moving average (EMA) of the online network weights. If the decay rate is set to 1, the target network never updates, effectively using a fixed random network. If the decay rate is set to 0, the target network updates every step along with the online network.
The authors tested different target network update rates. The results show that:
- If the target network updates too quickly, the learning process becomes unstable because the learning target changes too rapidly.
- If the target network updates too slowly, learning progresses slowly, and the learned features are of poor quality.
- In this 300-epoch ablation, gives 72.5%, above 0.9 and 0.999. The main experiment instead starts from 0.996.
Ablation Studies - Part 2
The authors further explore the relationship between BYOL and other contrastive learning methods and analyze why BYOL outperforms SimCLR.
Starting from InfoNCE, they consider an extended objective. The equation below is written as a score to maximize:
Where:
-
: Temperature hyperparameter.
-
: Negative sample influence coefficient.
-
: Batch size.
-
: Augmented views in the batch, with and being different augmented versions of the same image.
-
: Similarity function between views, defined as:
Where:
- SimCLR:
- (no predictor used).
- (no target network used).
- (using negative samples).
- BYOL:
- (uses predictor).
- (uses target network).
- (does not use negative samples).
- SimCLR:
Based on the above setup, the experimental results are shown in the table below:
First, they test different values of to verify whether negative sample pairs are necessary:
- When without the predictor and target network, the objective corresponds to SimCLR; changing alone does not turn every architecture into SimCLR.
- When (no negative examples), only the BYOL variant with both target network and predictor learns effectively in this table.
Among the variants tested in this ablation, the BYOL combination maintains good performance without negatives. This is not a claim that no other method can do so.
Next, they add a target network to SimCLR. The accuracy in Table 5(b) rises from 69.4% to 70.9%, a 1.5-percentage-point increase under these settings.
Finally, adding only a predictor to SimCLR does not outperform SimCLR in the table. The authors believe that the predictor and target network together are important for avoiding collapse in this no-negative setting.
Conclusion
The innovation of BYOL lies in completely removing the need for negative sample pairs and using the combination of a target network and a predictor to avoid the collapse solution. On ImageNet, BYOL achieves state-of-the-art self-supervised learning results and is close to the supervised learning baseline. Additionally, BYOL outperforms many existing contrastive learning methods.