Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
- Publication
- CVPR 2018Conference
- First submitted to arXiv
- arXiv:1805.01978
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{contrastivelearninginstdisc2018,
title = {{Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination}},
author = {Zhirong Wu and Yuanjun Xiong and Stella Yu and Dahua Lin},
year = {2018},
eprint = {1805.01978},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1805.01978}
}
Seizing a break in development, I quickly dive into some papers related to contrastive learning.
Although I’ve already listened to the master class before, reading through it myself feels more solid.
Before the concept of contrastive learning became popular, metric learning had already been widely used in the industry. The most well-known application is face recognition, with no exceptions.
The core of metric learning lies in learning feature representations , which establish a measure between samples and :
For open-set identity matching, inference usually compares learned embeddings rather than relying on the training identities’ classifier. A linear classifier can still be trained on these features for an appropriate downstream task.
Defining the Problem
The supervised metric-learning examples above rely on identity or class labels.
With only unlabeled data, we need another signal for learning similarity. Metric learning itself is not restricted to supervised methods.
Thus, researchers have come up with various solutions:
Generative Models
Another route is to learn how to generate or reconstruct data. Autoencoders and VAEs use encoders and decoders; GANs train a generator against a discriminator and need not include an image-encoding and reconstruction path.
The advantage is that no labels are required; as long as images are available, training can begin. However, the downside is the high computational cost. As the resolution of the reconstructed image increases, the computational cost also increases.
Generation or reconstruction quality and recognition-feature quality require separate evaluation. Some GAN training can also suffer mode collapse. These limitations depend on the model, objective, and data.
Self-Supervised Structural Learning
Self-supervised learning leverages the internal structure of the data to create a prediction task and then trains the model.
Let’s look at some interesting examples:
-
Context Prediction
-
Color Restoration of Grayscale Images
-
Jigsaw Puzzle Solving Task
-
Filling In Missing Parts of an Image
-
Counting Objects
After seeing so many proxy tasks, they all seem reasonable, but their usefulness for semantic recognition and the choice of task still need experimental justification.
Since there’s no consensus, let’s create one. Inspired by metric learning, the authors propose a novel feature learning training method, aiming to work without artificial labels.
Solving the Problem
To obtain a "good" feature representation, the authors adopt an "instance-level discrimination" strategy: treating each image as an independent "class" and training a classifier to determine whether each image belongs to its own "class."
Let’s take a look at the model architecture.
Model Architecture
As shown in the diagram above, on the left is a deep convolutional neural network labeled "Backbone CNN." Its function is to transform the input image (e.g., a 224224 image) into higher-level semantic features. For common CNN networks, this might involve several layers of convolution, batch normalization, pooling, and other operations. The final layer typically outputs a high-dimensional vector, which serves as the "visual feature representation."
The high-level features output from the Backbone CNN are usually still of large dimensionality (e.g., 2048 dimensions). To run efficiently with large datasets, more compressed and abstract feature descriptions are needed, so the authors add a "Projection Head" to project the CNN output vector into a 128-dimensional vector space.
After projection, the 128-dimensional vector is then subjected to L2 Normalization (), which means each sample is constrained to unit length. This is especially useful when computing similarity because, at this point, the dot product represents the cosine of the angle between the two vectors on the hypersphere, with values ranging from . This makes the calculation more intuitive and stable.
The Problem with Traditional Classifiers
In traditional Softmax classification models, each class corresponds to a weight vector . Suppose we have images (equivalent to "classes"), with corresponding weight vectors . For a given image , the feature vector produced by the neural network is . The conditional probability of the image being classified as the -th image (i.e., the -th "class") can be written as:
In this equation, is used to measure the alignment between the feature and class ; the denominator is the sum of the exponentiated values of all class comparisons (from 1 to ), ensuring that the total probability across all classes sums to 1.
In typical classification problems, the number of classes is fixed and not too large, so this parameterized Softmax is commonly used. However, here, since "each image" is treated as a new class, the number of classes can be enormous (tens of thousands, millions, or more). As a result, each class requires an independent weight vector , which not only requires significant storage space but also results in high computational costs.
Simply put, it becomes too computationally expensive.
Non-Parametric Classifier
To address the above problem, the authors propose a "non-parametric" approach:
- Replace the formula's with the feature vector stored in a memory bank.
In other words, the "class weight" for the -th image is no longer an independent , but instead, it directly uses the feature vector of the -th image itself. Additionally, the authors enforce normalization of these feature vectors to unit vectors (i.e., length 1) for ease of similarity calculation.
Thus, the Softmax formula can be rewritten as:
Here, is the "temperature parameter," which can adjust the "sharpness" of the Softmax distribution; is the feature vector of the current image produced by forward propagation; is the feature vector of the -th image stored in the "memory bank" (also a unit vector), and is the dot product of the two unit vectors, ranging from -1 to 1, which represents their similarity.
Memory Bank
The training objective for the non-parametric softmax is similar to that of typical classification problems: to maximize the probability of each training image being "correctly identified as itself," or equivalently, to minimize the negative log-likelihood over the entire training set:
Here, is denoted as . When computing , theoretically, the features of all images are needed. However, if every backpropagation requires passing all images through the neural network, the computational cost would be very high.
Thus, in practice, the authors maintain a "Memory Bank" () holding each image's most recently computed feature. It starts with random unit vectors, and after each training iteration, the entries for the images in that iteration are updated using their current forward-pass features .
Since each image has a corresponding position in memory, the previous features can be quickly retrieved to reduce computational cost when dealing with large datasets. If the total number of images is large, calculating the denominator would be prohibitively expensive. For every image, this computation would be , and when reaches millions or more, it becomes almost unmanageable.
Noise-Contrastive Estimation
The core idea of NCE is:
- Transform a multi-class classification problem into multiple binary classification problems.
- The goal of the binary classification: distinguish between "real data samples" and "noise samples."
In the multi-class scenario, each image is treated as a "class." With NCE, for a given image, we only need to distinguish "whether it is this image" versus "not this image (noise)."
The approach is to treat "other images" as noise, or sample negative examples from a noise distribution (), and let the model learn to differentiate between positive and negative examples.
For example, to calculate the "probability that feature comes from the -th image," we can write:
Where:
- is the current forward-pass feature of image ; is a candidate feature retrieved from the memory bank.
- is the normalization constant: with fixed, it makes probabilities over the candidate memory features sum to one. The paper's notation uses differently from the preceding softmax section.
- Computing all terms is costly for a large dataset.
The cleverness of NCE is that, in the case of "a large number of possible classes," it does not directly compute the full denominator of the formula. Instead, it separates the "positive sample" corresponding to the "class " and the "negative sample" corresponding to the "noise distribution."
Let the noise distribution be uniform ().
The paper introduces a hyperparameter , which represents that "there are times more noise samples than real samples," and then defines a "posterior probability":
The meaning of is that, given feature and class , it is the probability that " comes from the -th image (real data), rather than from noise." is the output of the above formula, and is the relative probability of selecting a noise sample corresponding to class .
The next key point is:
- Take the negative log-likelihood for positive samples (positive samples are correctly classified as positive).
- Take the negative log-likelihood for negative samples (negative samples are correctly classified as negative).
Thus, we have:
- Where represents the "real data distribution," i.e., the actual images (positive examples).
- represents the "noise distribution," i.e., the noise samples (negative examples).
- and are feature vectors retrieved from the Memory Bank (); one corresponds to the "positive sample" (), and the other is a randomly selected noise sample.
Intuitively, a larger means the model is more confident that " belongs to the -th image" (good); whereas a larger means the model correctly rejects the noise.
Approximation of
Although we have discussed many aspects, the biggest challenge remains unsolved.
We still need to calculate , which involves , and when is large, this is still a bottleneck.
A common approach in NCE is to treat as a "constant" for estimation or approximation, and not involve it in gradient updates. In this paper, the authors refer to the following work:
They use Monte Carlo approximation to estimate:
Here, is a set of randomly selected indices used to approximate the average value of the entire set.
Instead of summing all terms to calculate , now we only sample negative examples, perform the exponential operation on them, sum them up, and then multiply by the factor to approximate the full sum.
The paper also mentions that experimentally, a good estimate can be obtained with just a few batches early in training, and this estimate can be fixed for the remainder of the training, which works well.
Proximal Regularization
Unlike typical classification problems, here each "class" has only one training sample. Each class is visited just once per epoch, and the authors observe that the original training objective oscillates more and converges slowly.
To mitigate this instability caused by random sampling, the authors introduce Proximal Regularization. A smoothness constraint penalizes a large change from the image's stored feature at its last update. This helps stabilize training and accelerate convergence.
Specifically, we assume two variables:
- Current iteration (iteration ): For training data , compute the feature vector .
- Memory Bank: Stores the feature from the last time this image was encountered, denoted here as ; this need not be the immediately preceding global training iteration.
In the original NCE loss, an additional term is added.
Thus, the objective function becomes:
- The first term () is the original positive example loss in NCE.
- The second term () keeps the new feature close to that image's previously stored feature .
Through this proximal term, if the feature vector changes too drastically during each update, it will be penalized. As training progresses and the model converges, the ideal scenario is , and the penalty term gradually diminishes, eventually returning to the original objective function.
After incorporating this, the authors present the final objective function with Proximal Regularization:
This version retains the original NCE components for "positive example " and "negative example " while adding the term to suppress drastic changes in feature vectors.
To verify the effectiveness of Proximal Regularization, the authors compare the impact of different values (such as 0, 10, 30, 50) on training:
As shown in the figure above, when (i.e., no proximal term), the training loss oscillates more and falls more slowly. With an appropriately sized , its curve is smoother. The authors also report better learned features.
At test time, the authors find the nearest neighbors by cosine similarity in the feature space. They predict a class by weighting each neighbor's vote with , using as in training.
Discussion
Parametric vs Non-Parametric Softmax
Since the core of this paper is the "non-parametric Softmax," the authors first conducted an experiment on CIFAR-10 to compare the performance of "parametric Softmax" and "non-parametric Softmax."
Since the CIFAR-10 dataset is relatively small, the non-parametric Softmax denominator can be directly computed, allowing a direct comparison of the two approaches. The evaluation methods used were Linear SVM and k-NN classifiers, applied to "parametric" and "non-parametric" features, and their accuracy rates were compared.
The results, shown in the table above, reveal that the parametric Softmax achieved accuracy rates of 60.3% for SVM and 63.0% for k-NN, while the non-parametric Softmax achieved 75.4% and 80.8%, respectively. This demonstrates a significant improvement in performance with the non-parametric Softmax, validating the authors' approach.
Additionally, the authors explored the effects of using NCE approximation in the non-parametric setting, as shown in the table. The hyperparameter in NCE indicates how many negative samples are drawn for each positive sample:
- When (only 1 negative sample), the k-NN accuracy dropped drastically to 42.5%, indicating excessive approximation (too few negative examples).
- As increases, k-NN accuracy rises from 42.5% to 80.4% at , close to the full non-parametric Softmax's 80.8%. The same row's linear SVM result, 70.2%, is still below the full version's 75.4%. The mentioned in the prose means comparing with all the other CIFAR-10 training images, not an additional row in this table.
Performance in Image Classification Tasks
The authors then conducted larger-scale experiments on ImageNet to compare the performance of different methods across various network architectures.
The experimental setup was as follows:
- Dataset: ImageNet, approximately 1.28 million images, 1,000 categories.
- Temperature parameter : Set to 0.07.
- NCE negative samples : 4,096. The authors balanced "computational cost" and "feature quality."
- Training:
- Trained for 200 epochs using Momentum SGD.
- Batch size = 256.
- Initial learning rate of 0.03, maintained for the first 120 epochs, then decayed every 40 epochs by a factor of 0.1.
The authors listed several representative unsupervised (or self-supervised) learning methods, including:
- Random initialization (random initialization, as a lower bound).
- Self-supervised
- Adversarial learning
- Exemplar CNN
- Split-brain autoencoder: One of the recent self-supervised benchmark methods.
The authors used several backbones—AlexNet, VGG16, ResNet-18, and ResNet-50—to see how their method changes with architecture. Most other methods in the table come from separate papers, with different backbones or evaluation settings.
The evaluation methods were as follows:
- Linear SVM (conv1 ~ conv5): Trained a linear classifier on the intermediate features from different convolutional layers (such as conv1, conv2, conv3, ...) and tested the classification performance on the ImageNet validation set.
- k-NN (final 128-dimensional output): Directly applied nearest neighbor classification on the final 128-dimensional feature output.
The experimental results are shown in the table below:
With AlexNet, their conv5 features reach 35.6% top-1 accuracy with a linear classifier, above the other AlexNet results listed in the table.
With ResNet-50, their conv5 linear result reaches 54.0%; the listed Exemplar CNN result is 31.5% with ResNet-101. Their training and evaluation conditions differ. Across the four backbones used for this method, the conv5 value rises with architecture size.
The final 128-dimensional features yield k-NN accuracies of 31.3%, 33.9%, 41.0%, and 46.5% from AlexNet through ResNet-50. Each is below its own conv5 linear result, but the features still support classification. In this table, the method's later convolutional features are generally more useful to a linear classifier than the earlier ones.
Finally, some methods in the table use around 10,000-dimensional intermediate features. This method's final embedding has only 128 dimensions. The authors report about 600 MB to store features for all 1.28 million ImageNet training images and about 20 ms per image for a full-bank nearest-neighbor search on a Titan X GPU.
Cross-Dataset Generalization
The authors then tested the features learned on ImageNet, directly applied to another large dataset, Places (2.45 million images, 205 categories), without fine-tuning, only feature extraction, and then trained a linear classifier or performed k-NN on Places.
In the table above, ResNet-50 reaches 45.5% with a linear classifier on conv5 and 41.6% with k-NN on the final embedding. These exceed the listed methods in the corresponding columns, though the backbones and evaluated features differ across rows.
These results demonstrate that the features learned on ImageNet can maintain good performance even in different domains (scene classification), showing the method's cross-domain generalization ability.
Ablation Experiment - Feature Dimension
After confirming that the model architecture is effective, the authors conducted some ablation experiments.
The first explores the impact of feature dimension. The authors compared feature dimensions of 32, 64, 128, and 256, with results shown in the table above.
For ResNet-18, k-NN top-1 accuracy rises from 34.0% at 32 dimensions to 41.0% at 128 dimensions, then falls to 40.1% at 256 dimensions. In this tested setup, 128 dimensions give the best result in the table.
Ablation Experiment - Training Set Size
Next, the authors explored the impact of "training set size" on model performance. The experimental setup was as follows:
- ImageNet training set: Used 0.1%, 1%, 10%, 30%, and 100% of the images for feature learning.
- Evaluation: Used nearest-neighbor classification on the full labeled set to evaluate the learned features.
As the training share rises from 0.1% to 100%, accuracy in the table rises from 3.9% to 41.0%. Within this tested range, more unlabeled training images improve the learned features.
Visualization Analysis
Finally, the authors presented some feature visualization results, which demonstrate the "image search by image" functionality. By comparing image features, the model finds the closest images to the "query."
The experimental results are shown in the figure above:
- Best case (top four rows): The top 10 retrieved images are all from the same category as the query image (selected successful examples).
- Worst case (bottom four rows): None of the top 10 is in the query's true class. Some retrieved images still look similar in shape, color, or background; for others, the relationship is hard to judge from the figure.
The authors use these examples to suggest that the instance-level embedding can capture visual similarity beyond class labels. The figure provides qualitative examples.
Conclusion
The paper also includes experiments applied to "semi-supervised classification" and "object detection," but we won't go into detail here. Interested readers can refer to the original paper.
Overall, these experiments thoroughly validate the versatility and scalability of the authors' approach: it is effective not only for classification and retrieval but also demonstrates good generalization across different network architectures, data scales, and downstream tasks (semi-supervised classification, detection).
After the paper's release, the unsupervised learning framework based on NCE loss quickly achieved impressive results on large-scale datasets and demonstrated strong performance in practical applications.
There are still several classic papers to follow up on, so let’s continue reading.