Back to paper library
arXiv 2014IEEE TPAMI 2016Contrastive learningPaper notes

Exemplar CNN

Finding Invariant Features

Original paper

Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks

Paper authors

Alexey Dosovitskiy · Philipp Fischer · Jost Tobias Springenberg · Martin Riedmiller · Thomas Brox

First submitted to arXiv
arXiv:1406.6909
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{contrastivelearningexamplarcnn2014,
  title = {{Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks}},
  author = {Alexey Dosovitskiy and Philipp Fischer and Jost Tobias Springenberg and Martin Riedmiller and Thomas Brox},
  year = {2014},
  eprint = {1406.6909},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1406.6909}
}
On this page

While reading the paper, I noticed that many studies reference the ideas from this work, so let's take some time to look at it.

The concept of the paper is simple, but many experimental results can inspire us with new ideas.

Defining the Problem

The advantage of supervised learning is also its drawback: it requires a large amount of labeled data. In some specific application scenarios, such data might be hard to obtain or expensive.

Moreover, if we want to perform transfer learning between different tasks, such as switching from object classification to descriptor matching, the performance of supervised learning might not be ideal.

Since it’s not ideal, a simple idea arises: what if we don’t use data labels?

But if we don't use data labels, how should we train the model?

The authors suggest: we need a proxy task.

Solving the Problem

The authors propose a training method that does not require manual annotations, called Exemplar-CNN.

The core concept is very simple:

  1. Randomly extract image patches: To focus more on areas with rich object information, image patches of size 32×3232 \times 32 are sampled from unlabeled images with probability proportional to the mean squared image-gradient magnitude within each patch.
  1. Apply various random transformations to the image patches: Such as translation, scaling, rotation, color adjustment, contrast adjustment, etc., to generate multiple "deformed versions." These transformations encourage the network to learn the corresponding invariances (e.g., rotation invariance, color invariance).
  2. Form "proxy categories" and perform discriminative learning: Each original image patch and its deformed versions are considered as "positive samples" in the same proxy category.

The system repeats this process across many images, generating many proxy categories, and trains a CNN to distinguish "which image patches (including all deformed versions) belong to the same proxy category and which belong to other proxy categories." This way, the model learns to distinguish different image patches and is encouraged to become less sensitive to "various deformations of the same image."

The list of transformations used in the paper is as follows:

  • Translation: Translations in the vertical or horizontal direction, not exceeding 20% of the image patch size.
  • Scaling: Rescale the image patch proportionally, with a scale factor between 0.7 and 1.4.
  • Rotation: Rotation angle within ±20 degrees.
  • Contrast 1: Multiply each pixel’s projection in all principal component directions by a factor between 0.5 and 2 (independent for each principal direction, the same for all pixels in the patch).
  • Contrast 2:
    • Raise the saturation and brightness (S and V components in HSV color space) to a power between 0.25 and 4.
    • Then multiply by a factor between 0.7 and 1.4, and add an offset between −0.1 and 0.1.
  • Color: Apply an offset between −0.1 and 0.1 to the hue (H component in HSV color space).

The training data after cropping looks like this (using STL-10 as an example):

Experimental Setup

To compare Exemplar-CNN with previous unsupervised feature learning methods, the authors performed a series of experiments using the following datasets: STL-10, CIFAR-10, Caltech-101, and Caltech-256.

Except for the dataset-transfer experiment, proxy-class training patches come from STL-10's unlabeled images. For classification evaluation, a linear SVM is then trained using labeled examples from each dataset.

Details about the datasets, such as the amount of data, number of categories, image sizes, are as follows:

  • Number of categories:
    • STL-10 and CIFAR-10: 10 categories.
    • Caltech-101: 101 categories.
    • Caltech-256: 256 categories.
  • Image size processing:
    • CIFAR-10: Adjusted from 32×3232 \times 32 to 64×6464 \times 64.
    • Caltech-101: Adjusted to 150×150150 \times 150.
    • Caltech-256: Adjusted to 256×256256 \times 256.

Next, the authors designed several network architectures as follows:

  • Small network (64c5-64c5-128f):
    • Two convolutional layers (64 filters each).
    • One fully connected layer (128 units).
    • Final Softmax layer.
  • Medium network (64c5-128c5-256c5-512f):
    • Three convolutional layers (64, 128, 256 filters).
    • One fully connected layer (512 units).
  • Large network (92c5-256c5-512c5-1024f):
    • Three convolutional layers (92, 256, 512 filters).
    • One fully connected layer (1024 units).
  • Convolution and pooling details:
    • All convolution filters correspond to 5×55 \times 5 regions of the input.
    • 2×22 \times 2 max pooling is applied after the first two convolutional layers.
  • Other techniques:
    • Dropout is used in the fully connected layers.
    • Based on the Caffe implementation, training details are in Appendix B.2.

The testing method applies the network to images of any size, computes feature maps, and selects the appropriate pooling method based on the dataset:

  • STL-10 and CIFAR-10: Quadrant max pooling (4 values per feature map).
  • Caltech-101 and Caltech-256: Three-level spatial pyramid pooling (1+4+16=211+4+16=21 values).

Finally, the learned features are frozen and a one-vs-rest linear SVM is trained on labeled examples for classification. The feature pretraining is unsupervised; this downstream evaluation still uses class labels.

Discussion

Comparison with Other Methods

The table above compares the performance of Exemplar-CNN with other unsupervised feature learning methods on different datasets.

The larger Exemplar-CNN has the best result among the unsupervised feature-learning methods shown for each dataset. On STL-10, its 74.2% exceeds the table's supervised reference of 70.1%. The supervised results are higher on the other datasets, and the methods use different training conditions.

Ablation - Number of Proxy Categories

The authors first tested the impact of the number of proxy categories on performance.

The experimental design adjusted the number of proxy categories NN from 50 to 32,000 and observed changes in classification accuracy.

In the figure, STL-10 accuracy rises until roughly 4,000–8,000 surrogate classes, then levels off or declines, while validation error on the surrogate-class task keeps rising. The authors suggest that too many similar or duplicate seed patches become difficult to distinguish and can hurt the usefulness of the learned features.

Ablation - Number of Samples

Next, the authors explored the effect of the number of samples per proxy category KK.

They adjusted KK from 1 to 300 and observed changes in classification accuracy.

STL-10 accuracy improves with more samples per surrogate class and levels off near 100. The authors take this to mean that about 100 transformed samples suffice to approximate the expected objective in their Equation (3) in this experiment; further samples change the result little here.

Ablation - Type of Image Augmentation

The authors further explored the effect of different types of image augmentations on performance.

The experimental design fixed the original "seed" samples and adjusted the types of transformations applied to the proxy data, including scaling, rotation, translation, color change, contrast change, and so on.

The negative values in the figure are the accuracy differences after removing each type of transformation, relative to using all of them. Removing translation, color variation, or contrast variation usually causes a larger drop; scaling and rotation have smaller effects. Sensitivity also differs among the datasets.

Ablation - Parameter Configuration

The table above shows the performance of Exemplar-CNN across different datasets.

Some wider or deeper configurations improve accuracy, but a larger network is not always better: the four-convolution-layer 64c5-64c5-64c5-64c5-128f performs worse than the three-layer 64c5-64c5-64c5-128f. Increasing surrogate classes from 8,000 to 16,000 for the same architecture also has mixed effects across the evaluation columns.

Filter Visualization

The figure above shows the filters learned by the first layer of the 64c5-64c5-128f network during training on proxy data from different datasets, arranged as follows:

  • Top: Proxy data from the STL-10 dataset.
  • Middle: Proxy data from the CIFAR-10 dataset.
  • Bottom: Proxy data from the Caltech-101 dataset.

The first-layer filters trained from different datasets vary in their color patches and local texture.

The authors separately compare training data sources in classification experiments: patches from the target dataset generally work best, but the differences across datasets are small. Those results support feature transfer more directly than the filter display alone.

Conclusion

This paper presents an unsupervised feature learning method based on a discriminative objective that does not rely on object category labels. The authors generate proxy labels through data augmentation and define the invariances the network must learn via image transformations.

Experimental results demonstrate that Exemplar-CNN significantly improves feature learning for classification accuracy, and the discriminative objective offers advantages over previous unsupervised learning objectives.