Exploring Simple Siamese Representation Learning
- Publication
- CVPR 2021Conference
- First submitted to arXiv
- arXiv:2011.10566
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{contrastivelearningsimsiam2020,
title = {{Exploring Simple Siamese Representation Learning}},
author = {Xinlei Chen and Kaiming He},
year = {2020},
eprint = {2011.10566},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2011.10566}
}
Although we often refer to it as contrastive learning, this field should actually be called representation learning, and "contrastive" learning is simply one widely used approach.
In addition to contrastive learning, there are other learning methods, such as "self-prediction" or "cluster centers," which continue to push the performance of representation learning. However, the field still seems to lack consensus on what the key factors influencing representation learning are.
This paper is quite rich in content, so let's take a closer look.
Defining the Problem
Some of the recent methods have adopted a Siamese network architecture to address the issue of representation learning.
However, Siamese networks are prone to collapse, where the outputs converge to a constant value.
To solve the collapse problem, past research has proposed several approaches:
- Contrastive Learning: Avoids collapse by using negative samples.
- Clustering: Uses assignment constraints to prevent collapse, creating competition among samples without necessarily defining explicit negative pairs.
- Momentum Encoder: Uses a slowly updated target branch as part of a design that avoids collapse, as in BYOL.
So, is it possible to avoid collapse effectively without using any of these strategies (negative samples, clustering, momentum encoder)?
Solving the Problem
The method proposed in this paper is called SimSiam, short for Simple Siamese Network.
Its goal is to perform unsupervised representation learning with the simplest Siamese network architecture:
First, a single input image undergoes two random augmentations to create two different views: .
These two views are fed into a shared-weight encoder , which consists of two sub-networks:
- Backbone: For example, ResNet-50.
- Projection Network (Projection MLP): A multilayer perceptron that maps the output of the backbone into the latent space.
Then, the output of one view is passed through a Prediction Network (Prediction MLP) to have the two views predict each other.
The evaluation is done using negative cosine similarity:
- Let , the output vector after passing through the prediction network .
- Let , the output vector of the other view after passing through the encoder .
The loss function is defined as:
- represents the norm of a vector (i.e., its Euclidean length).
The greater the cosine similarity, the more similar the two vectors are, so the authors minimize the negative cosine similarity (which is equivalent to maximizing cosine similarity). This expression is equivalent to the mean squared error (MSE) between normalized vectors (differing by a positive scale factor and an additive constant).
To allow the two views to predict each other symmetrically, the final loss function is defined as the average of the negative cosine similarity in both directions:
The theoretical minimum of this loss function is -1, meaning the two vectors are perfectly aligned.
At this point, there isn't much difference from previous research, and the core concept of this paper is introduced next.
Stop Gradient
To prevent the model from collapsing during training, the key technique proposed by the authors is:
- Stop gradient operation!
In practice, the equation is rewritten as:
means that in this equation, the vector is treated as a constant and does not propagate gradients back to the encoder .
The same method is applied in the other direction, and the final loss function is written as:
The encoder weights are shared. In each directional term, gradients flow only through the prediction path; the stopped target path receives none. That view still contributes gradients as the prediction path in the other term. This avoids collapse in the reported setup, rather than being an unconditional guarantee.
The authors provide detailed implementation methods in the paper:
Discussion
Why? Why is it that such a simple method can make the model converge?
Let's first look at the results briefly, as the authors provide extensive derivations and arguments later on.
Comparison with Other Methods
The authors compare SimSiam with other mainstream unsupervised representation learning methods (SimCLR, MoCo v2, SwAV, BYOL). The comparison conditions include:
- Using the standard ResNet-50 architecture.
- Pretraining is done on ImageNet with image size , and each method uses two different views.
- Linear classification is used as a metric to evaluate the quality of representation learning.
The authors emphasize that, for fair comparison:
- The results for the other methods are reproductions by these authors; SimSiam is their method.
- SimCLR, SwAV, and MoCo v2 are slightly improved by the authors (e.g., increasing the number of layers in the projection MLP and adopting a symmetric loss function), and these improvements are marked as "+" in the results.
- Batch sizes are not matched: SimSiam and MoCo v2 use 256, while the other three methods use 4096.
In the 100-epoch comparison, SimSiam's 68.1% is the highest in the table, ahead of SimCLR, MoCo v2, SwAV, and BYOL. The 67.7% value belongs to a different baseline setting. At 200 epochs, BYOL reaches 70.6%, above SimSiam's 70.0%; at 400 epochs, SimSiam reaches 70.8%, still close to MoCo v2 and SwAV.
Despite its simplicity, SimSiam performs very well at short and medium training lengths.
Exploration of Basic Architecture Design
To compare various architectures in more detail, the authors provide model architecture diagrams for different setups:
-
Comparison with SimCLR
In the ImageNet linear-evaluation settings shown, SimSiam exceeds the authors' reproduced SimCLR at every training length. The methods still differ in their predictor, stop-gradient, and batch size.
In this paper, SimSiam can be viewed as "SimCLR without negative samples" as a conceptual architecture comparison. The experiment shows that the complete SimSiam setup learns useful representations without negatives and does not collapse.
Therefore, negative samples are not a necessary design for good performance in representation learning.
-
Comparison with BYOL
BYOL uses a momentum encoder (inherited from the MoCo series), whereas SimSiam does not.
In the experiment, BYOL scores above SimSiam at 200, 400, and 800 epochs; SimSiam is higher in the 100-epoch comparison.
In this paper, SimSiam can be seen as "BYOL without a momentum encoder", which suggests that a momentum encoder is not necessary for preventing model collapse.
This experiment shows that a momentum encoder is not required for SimSiam to avoid collapse; the two implementations are not otherwise identical.
-
Comparison with SwAV
SwAV essentially uses online clustering and Sinkhorn-Knopp (SK) transformations.
In this paper, SimSiam can be viewed as "SwAV without online clustering" after abstracting away components such as the prototype layer. SwAV is below SimSiam at 200 and 400 epochs in the table, but higher at 800 epochs.
Thus, the complete SimSiam setup avoids collapse without online clustering; the accuracy differences in this figure cannot be attributed to clustering alone.
Ablation Studies
-
Necessity of Stop-Gradient
In the same model architecture and hyperparameter settings, one group uses stop-gradient, while the other completely removes it.
The experimental results show that without stop-gradient, the model quickly collapses, and the training loss reaches the theoretical minimum of -1, with the standard deviation of the output vector approaching zero, indicating that the model has collapsed into a constant vector.
With stop-gradient, the per-channel standard deviation of normalized features approaches , a reference value for normalized zero-mean isotropic Gaussian vectors. This argues against a constant-vector collapse.
Therefore, the network architecture design alone (predictor, BN, normalization) is insufficient to prevent collapse, while stop-gradient is a crucial design.
-
Analysis of the Role of the Predictor ()
The authors further analyze the role of the predictor network in the model.
The experiment shows that when the predictor is removed (i.e., set as an identity map), the model immediately collapses, because the symmetric loss function combined with the identity map causes stop-gradient to lose its effect.
If the predictor is randomly initialized and then fixed, linear evaluation reaches only 1.5%. The authors observe non-convergence and high loss in this run, rather than diagnosing collapse from that number alone. The predictor must adapt as the representations are learned.
Therefore, the predictor network is an essential component of SimSiam, and must be trained along with the encoder.
-
Analysis of Batch Size Impact
The authors investigate the model's performance across different batch sizes, ranging from 64 to 4096, with linear learning rate adjustments. All experiments use SGD (not using the LARS optimizer).
Accuracy ranges from 67.9% to 68.1% for batch sizes 256–2048; it falls to 66.1% at 64 and 64.0% at 4096. Under this SGD setting, SimSiam does not need a batch of 4096, but its performance is not identical at every batch size.
-
Analysis of Batch Normalization (BN) Impact
The authors test the impact of BN on the model.
The experimental results are as follows:
- (a) Completely removing BN from the MLP: no collapse, but poor performance (34.6%) and difficult optimization.
- (b) Adding BN to the hidden layers: significant improvement to 67.4%.
- (c) Adding BN to the projection MLP output layer: further improvement to 68.1% (best setting).
- (d) Adding BN to the prediction MLP output layer: training instability and oscillation.
BN helps optimization in these experiments. The authors did not observe that BN alone prevents collapse: both runs in the stop-gradient ablation used the same BN configuration, yet the run without stop-gradient collapsed.
-
Analysis of Replacing Similarity Functions
The authors replace cosine similarity with cross-entropy similarity to examine the model's stability.
Experimental results:
- Cosine similarity: 68.1%
- Cross-entropy similarity: 63.2%
- Both methods prevent collapse, but cosine similarity performs better.
This replacement shows that avoiding collapse does not depend only on cosine similarity, although the choice of similarity function still affects accuracy.
-
Symmetry of Loss Function (Symmetrization) Analysis
The authors investigate whether the loss function needs to be symmetric.
Experimental results:
- Symmetric version (sym): 68.1%
- Asymmetric version (asym): 64.8%
- Asymmetric version with doubled sampling (asym 2×): 67.3%
The conclusion is that symmetric loss functions positively impact model performance but are not a necessary condition to prevent model collapse.
Expectation-Maximization Hypothesis
Through experiments, the authors found that the core design of SimSiam, the "stop-gradient," effectively prevents model collapse and leads to excellent representation learning results. But why is this simple operation so effective? What underlying principle is hidden behind it?
To explain the mechanism, the authors propose a key hypothesis:
SimSiam implicitly solves an optimization problem similar to "Expectation-Maximization (EM)."
Mathematical Representation
To build the EM-like hypothesis, the authors consider the following auxiliary optimization problem. It is an explanatory model rather than an exact equivalence for the full SimSiam algorithm:
Where:
- : is the encoder network determined by parameters .
- : represents the image after random data augmentation.
- : is a set of additional optimization variables, where each image has a corresponding representation vector .
- : can intuitively be understood as the "feature representation" of image , but it is not directly output by the neural network; it is instead a variable optimized through the process.
The auxiliary problem is:
This optimization problem is similar to the classic "K-means clustering" problem:
- is the encoder’s parameter vector.
- is a continuous target representation for image .
The analogy to K-means concerns alternating updates; these variables are not literally cluster centers and discrete class assignments.
This problem can be solved using "alternating optimization":
- Fix , optimize :
In this step, since is fixed as a constant, gradients will not propagate back to , which naturally leads to the stop-gradient operation.
- Fix , optimize :
This step is independent for each image , and the optimal solution is to set the representation for each image equal to the average feature of the image after all augmentations:
SimSiam's EM Approximation
Under this hypothesis, replacing the augmentation expectation with a sampled view gives a Siamese update:
- The expectation, which should be averaged, is approximated by using only a single image augmentation , i.e.:
- This approximated representation is then fed back into the original optimization problem, yielding a related update without the predictor (whose role is considered next):
This form directly reflects the Siamese network architecture and naturally introduces the stop-gradient operation.
The True Role of the Predictor
The aforementioned EM hypothesis does not explain the necessity of the predictor , so the authors further hypothesize:
- The function of the predictor network is actually to approximate the expectation of another image augmentation:
Since it is not feasible to directly compute the expectation over all image augmentations, the predictor network is trained to learn and approximate this expectation. Therefore, the predictor essentially serves as an approximation tool to make up for the lack of expectation computation.
The Effect of Symmetric Loss Function
The authors also note that the above EM hypothesis does not require the use of a symmetric loss function (symmetrized loss). The effect of symmetrization is essentially like taking an additional pair of image augmentations at each optimization step, which helps more accurately approximate the expectation and can improve model accuracy. However, it is not a necessary condition to prevent collapse.
Experimental Validation of the Hypothesis
The authors conducted two concept verification experiments to support the above EM hypothesis:
-
Multi-step Alternating Updates
The experiment found that increasing the number of alternating optimization steps (e.g., 10 steps, 100 steps) still yields good results, even better than the original SimSiam:
- 1-step (SimSiam): 68.1%
- 10-step: 68.7%
- 100-step: 68.9%
- 1-epoch: 67.0%
This supports the effectiveness of EM alternating optimization.
-
Verification of Image Augmentation Expectation Approximation
By using a moving average to approximate the expectation, even without the predictor network , reasonable performance (55.0%) can still be achieved. If the moving average is not used and the predictor is removed, the model collapses completely (0.1%).
This supports the expectation-approximation hypothesis.
While the authors have proposed a reasonable EM hypothesis, they have not fully explained why the model collapse can be prevented. They speculate that it may be related to the initialization and path of the alternating optimization, but this remains an intuitive level of explanation and requires further theoretical exploration in the future.
Conclusion
In this paper, the authors explored the "extremely simple design of Siamese networks," and despite the simplicity of the SimSiam architecture (no negative samples, no clustering, no momentum encoder), it still performs very competitively across various experiments.
This suggests that the success of recent mainstream self-supervised methods (such as SimCLR, MoCo, SwAV, BYOL) may not stem from the various additional complex designs but fundamentally from their "shared Siamese network architecture itself"!
The minimalist Siamese network architecture might be the real core reason behind the success of recent representation learning, warranting further exploration and research.