Back to paper library
arXiv 2021ICML 2021Multimodal learningPaper notes

CLIP

Breaking the Dimensional Barrier

Original paper

Learning Transferable Visual Models From Natural Language Supervision

Paper authors

Alec Radford · Jong Wook Kim · Chris Hallacy …

Show all 12 authors

Alec Radford · Jong Wook Kim · Chris Hallacy · Aditya Ramesh · Gabriel Goh · Sandhini Agarwal · Girish Sastry · Amanda Askell · Pamela Mishkin · Jack Clark · Gretchen Krueger · Ilya Sutskever

Publication
ICML 2021Conference
First submitted to arXiv
arXiv:2103.00020
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityclip2021,
  title = {{Learning Transferable Visual Models From Natural Language Supervision}},
  author = {Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever},
  year = {2021},
  eprint = {2103.00020},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.00020}
}
On this page

Try Describing This Image:

  • A cup of coffee... um, with a book underneath the cup? Both are on a table?
  • Is this a corner of a café in the morning?
  • Brown table, brown chairs, and brown coffee? (Come on, be serious!)
  • …

Imagine describing it with a story like this:

On a peaceful morning, sunlight softly filters through the gaps in the curtains, gently illuminating a simple wooden table. On the table rests a freshly brewed cup of coffee, and the aroma of the coffee mingles with the sunlight, evoking warmth and hope for the day ahead. The shadow of the cup stretches across the table, casting a long reflection, which, along with the green plants by the window, creates a beautiful scene. The surface of the coffee ripples slightly, as if whispering the calmness of the morning and the beauty of life. Beside the cup, an open book lies quietly, waiting to be read. This tranquil morning scene, with coffee, sunlight, greenery, and books, forms a warm and serene moment, telling a story of life’s simplicity and beauty.

That is an imagined description; the illustration's generation prompt has not been verified. It is not a figure from the CLIP paper, and CLIP itself does not generate images.

When trying to describe this image, you might realize that even ImageNet, with more than 20,000 categories in its larger hierarchy and millions of images, feels inadequate. The zero-shot benchmark below uses ImageNet's separate 1,000-class setting.

  • There are countless ways to describe the same image.

Defining the Problem

In contrast to the single hierarchical labels in ImageNet, humans can interpret an image from countless angles—through color, shape, emotions, and narrative context.

This highlights the distinction between unimodal learning and multimodal learning: the former processes information from a single perspective, while the latter integrates different types of information, providing a richer and more comprehensive interpretation.

Traditional datasets like ImageNet often ignore meaningful information, such as object relationships, context, or emotions evoked by an image.

The goal of CLIP is to break these limitations by learning from image–text pairs, then using natural language to define candidate classes for a new task. Here, zero-shot means using no labeled training examples from the target dataset; its class names or images may still occur in web pre-training.

So, how did they achieve this?

The Solution

Model Architecture

The above diagram illustrates the pre-training architecture of CLIP.

Consider a pair of image and text—such as a picture of a dog with the caption, “a cute puppy.”

In a training batch, CLIP receives multiple such pairs. The image encoder processes the images (via ResNet or ViT) to extract features, while the text encoder (using a Transformer) extracts textual features.

The model compares the features of all NN image–text pairs in the batch, forming an N×NN\times N matrix of cosine similarities. Cross-entropy in both directions—image to text and text to image—raises the scores of correct pairs (like the dog image and “a cute puppy”) relative to mismatched pairs (like the dog image and “an apple”). The encoders are trained from scratch; the Transformer does not produce captions here. At inference, candidate class descriptions become text vectors, and the model chooses among those supplied candidates.

What next?

Next, they fed 400 million image-text pairs into the model for training!

Large-Scale Dataset

Initially, the authors used datasets like MS-COCO, Visual Genome, and YFCC100M.

However, these datasets were too small for this purpose. MS-COCO and Visual Genome each offer around 100,000 training photos—little beside computer vision systems that process billions of Instagram photos. YFCC100M, though larger, has many low-quality descriptions, including automatically generated filenames or camera exposure settings. Filtering for usable English natural-language text also makes the available set much smaller.

The solution? Build a new large-scale dataset—WIT (WebImageText)—comprising 400 million image–text pairs collected from publicly available web sources.

Training Details

The authors trained a series of 5 ResNet and 3 Vision Transformer (ViT) models.

  • ResNet Series: They trained ResNet-50 and ResNet-101, followed by scaled versions like RN50x4, RN50x16, and RN50x64—each representing 4x, 16x, and 64x the computational power of ResNet-50.
  • ViT Models: The team trained ViT-B/32, ViT-B/16, and ViT-L/14. All eight models ran for 32 epochs with Adam, decoupled weight decay, and a cosine learning rate schedule. Initial settings came from search and manual adjustment on the ResNet-50 baseline, then were adapted heuristically for larger models.

The learnable temperature began at τ=0.07\tau=0.07, and logit scaling was capped at 100. Training used batches of 32,768 pairs, mixed precision, gradient checkpointing, half-precision Adam statistics, and stochastic rounding of the half-precision text-encoder weights to save memory and accelerate the run.

Training the largest ResNet model (RN50x64) required 592 V100 GPUs for 18 days, while ViT-L/14 needed 256 V100 GPUs for 12 days. ViT-L/14 received one additional pre-training epoch at 336-pixel resolution, designated ViT-L/14@336px.

Discussion

Zero-Shot Learning

Here, “zero-shot” means building a classifier from class names or descriptions without using labeled training examples from the target dataset. Those classes may still appear among the 400 million pre-training pairs. CLIP does deliver in some settings, but the details expose room for improvement:

  1. Strong results on particular datasets:

    Zero-shot CLIP beats a supervised logistic-regression baseline trained on ResNet-50 features on 16 of 27 datasets. Its lead on Stanford Cars and Food101, and its shortfall on Flowers102 and FGVC Aircraft, vary widely. That is task-dependent performance, not a victory over every fully supervised classifier.

  2. Matching a few-shot baseline:

    On the 20 comparable datasets, zero-shot CLIP averages roughly the same accuracy as a classifier trained on the same CLIP features with four labeled images per class. The comparison shows the value of making a classifier from text.

  3. The price of the equivalent few-shot transfer:

    Depending on the dataset, the authors estimate that a few-shot classifier needs anywhere from less than one to 184 labeled examples per class to match zero-shot CLIP. Those samples belong to the comparison classifier. Zero-shot CLIP itself does not suddenly require 184 examples.

  4. Comparison with a fully supervised classifier:

    If a fully supervised linear classifier is trained on CLIP features, zero-shot accuracy remains about 10–25 percentage points lower on most datasets. Only five datasets have gaps of no more than three points. That baseline differs from the earlier ResNet-50-feature comparison.

  5. A linear performance relationship:

    Across datasets, the fully supervised linear classifier on CLIP features and the zero-shot classifier have positively correlated results, with a coefficient of 0.82. Better representations seem related to better transfer; adding target labels does not itself improve a classifier that remains zero-shot.

  6. The scaling pattern:

    For the five CLIP models compared over roughly 44 times the training compute, average zero-shot error follows a fairly smooth trend on the paper's logarithmic compute plot. Individual tasks vary much more. The plot does not isolate model size from data diversity.

Are the Representations Any Good?

A closer look at the learned image features tells us more about both the strengths and the limits:

  1. Assessing feature quality:

    A linear classifier fitted on frozen CLIP image features measures whether those pre-trained features support the tested tasks.

  2. Why a linear classifier?:

    Linear probing makes the pre-trained representation easier to inspect. End-to-end fine-tuning changes that representation and might hide a capability the original model did not learn. This is why the authors use linear probes for this comparison.

  3. Scale and efficiency:

    CLIP's larger models, such as RN50x64 and the ViTs, show a relationship among model scale, feature quality, and compute efficiency. They also demand more computation, which can limit use on smaller devices.

  4. Different kinds of tasks:

    The features transfer to geolocation, OCR, facial emotion recognition, and action recognition, among other tasks. Results vary with the task and data distribution.

  5. A broader set of datasets:

    Evaluation on 27 datasets demonstrates substantial transfer. The authors also acknowledge that this set was selected progressively during development. Its choice may have adapted to the model, and a more systematic zero-shot benchmark is still needed.

  6. Self-supervised systems:

    The linear-probe comparisons include systems such as SimCLRv2. They compare transfer results under specified conditions, not the universal training efficiency of one method against another.

  7. Comparison with a strong model:

    With a linear probe, CLIP features outperform Noisy Student EfficientNet-L2 features on 21 of the 27 datasets. Noisy Student still has advantages on ImageNet, low-resolution datasets, and some specialized tasks. These are feature-plus-linear-classifier results, not zero-shot results.

Is CLIP Truly Generalizable?

Cross-domain performance matters because an ImageNet score alone tells us little about a change in image distribution. In the paper's Figure 13, the conventional ResNet-101 and zero-shot CLIP both reach 76.2% on ImageNet. The ResNet's accuracy drops sharply on several natural shifts, while CLIP retains more of its original performance. Their respective scores are 64.3% and 70.1% on ImageNetV2, 37.7% and 88.9% on ImageNet-R, and 32.6% and 72.3% on ObjectNet. The banana example at the right is an illustration at matched ImageNet accuracy, not a result for every class.

In the separate Figure 14 experiment, training a supervised linear classifier on CLIP features raises ImageNet accuracy by 9.2 percentage points to 85.4%, while mean accuracy on the tested shifts falls slightly. That classifier is different from Figure 13's zero-shot classifier. A better in-distribution score need not carry over to shifted data.

Human vs. Machine

To probe the difference between CLIP and people, the authors asked five participants to classify 3,669 Oxford-IIIT Pets test images into 37 cat and dog breeds. Each person could answer “I don't know.” They first saw no breed examples, then one and two examples per breed. The point was to see how quickly a person improves when shown a reference.

On a separate STL-10 task, human accuracy reached 94%; the 97–100% figures are attention-check accuracy, meant to show that participants were paying attention. For Oxford-IIIT Pets, the individual per-class average rose from 53.7% with no examples to 75.7% with one. Five-person majority vote rose from 57.0% to 80.3%, reaching 85.0% with two examples. A second example barely improved the individual average. The first example mainly helped on images about which people had been uncertain.

Zero-shot CLIP reached 93.5% on this Pets test, above the participants. Still, people could look directly at a breed example, whereas the paper's few-shot linear classifier on CLIP features did not use prior knowledge in the same way. This experiment does not give a like-for-like ranking of human and model few-shot learning.

Overlap in Data

Did the model peek at the answers? Training on a huge web corpus may unintentionally include downstream evaluation images, which could inflate a benchmark score without representing real transfer.

The duplicate analysis found no detected overlaps in some unusual or synthetic datasets, including MNIST, CLEVR, and GTSRB. Country211, built using public image sources, had an estimated 21.5% overlap. Even there, the paper estimates an overall accuracy difference of just 0.2 percentage points by comparing the full test set with the non-overlapping subset. The training text of a repeated image may not have contained the information needed for the downstream question.

But overlap analysis is subtle. On Kinetics-700, many detected “duplicates” were black transition frames, and the overlap subset performed about 20% worse, not better. On low-resolution CIFAR-100, a small bird or plane could create a false image match. Both class distribution and example difficulty can differ between overlap and clean subsets, hiding or mimicking overfitting.

Earlier large-scale pre-training studies by Mahajan et al. (2018) and Kolesnikov et al. (2019) also found small changes to aggregate performance; Kolesnikov and colleagues compared deduplication strategies. That context helps, but it does not settle CLIP's case. Its detector approached 100% accuracy on a proxy task and achieved high precision after manual checks and threshold adjustment, yet recall cannot readily be measured over all 400 million pairs. The black-frame and low-resolution examples illustrate challenges for leakage detection.

Limitations

CLIP's results are encouraging, but serious limitations remain:

  1. Performance gaps:

    Zero-shot CLIP can match a linear classifier on ResNet-50 features for some datasets, yet remains well below state-of-the-art models on many others. The authors estimate that reaching the leading benchmark level by scaling this approach alone could require about 1,000 times more compute, impractical on the hardware available in the paper.

  2. Learning and transfer across tasks:

    It struggles with fine-grained classes such as car models, flowers, and aircraft variants, and with more abstract, systematic tasks such as counting objects. A natural-language label does not make every visual problem easy.

  3. Generalization:

    Performance on several natural shifts is strong, but truly out-of-distribution data can still defeat it. Zero-shot CLIP reaches only 88% on MNIST handwritten digits, below even a logistic-regression baseline on raw pixels. It would be naive to assume that a sufficiently varied web corpus makes every future dataset effectively in-distribution.

  4. Data and compute efficiency:

    The paper's “405 years” is the time required to display the roughly 12.8 billion image presentations of 32 epochs serially at one image per second. It is not the actual training duration; the GPU-cluster runs above took days. Self-supervision and self-training suggest possible ways to improve data efficiency, but that remains a direction for research.

  5. Evaluation and optimization:

    Although the model is called zero-shot, the researchers repeatedly consulted full validation-set performance to guide development. Those sets often contain thousands of examples, which is unrealistic in a strict zero-shot use case; Oliver et al. (2018) raised related concerns for semi-supervised learning. Performance can also fall unexpectedly when moving from zero-shot to the paper's few-shot linear-classifier setting. That transition needs better methods.

  6. Social bias:

    Uncurated web images and text can encode social bias. A model that scores supplied descriptions can expose those biases differently as the candidate set and thresholds change.

Further work has to combine CLIP's useful zero-shot transfer with efficient learning from a few examples and more reliable evaluation.

Ethical Concerns

  • Bias:

    The authors used FairFace and other exploratory tests to examine differences among groups. With one candidate set containing non-human animal and crime-related words, some people were misclassified into those harmful labels at rates that varied by age, gender, and race. These rates belong to that particular label set, not every possible deployment.

    Adding a “child” candidate sharply reduced harmful assignments for images of people under 20. In a test on members of Congress, changing output thresholds altered the distribution of gender-linked labels. Thus the training corpus is only part of the story: candidate labels, wording, and thresholds shape how bias appears. The tests are preliminary.

  • Surveillance:

    The paper examines how a more general visual model could be used in surveillance, including low-resolution camera images and zero-shot celebrity identification. Using VIRAT and data from Varadarajan and Odobez, it assembled 515 frames from 12 videos. Coarse scene classification, such as an empty parking lot or school campus, reached 91.8% top-1. Adding similar distractor descriptions to this coarse-classification stress test dropped accuracy to 51.1%; 40.7% of answers were close but wrong. A separate fine-grained small-object test was near chance. The 51.1% figure is not its accuracy.

    On CelebA images of celebrities “in the wild,” zero-shot top-1 identity accuracy was 59.2% among 100 candidates and 43.3% among 1,000. These numbers are not competitive with purpose-built recognition systems, but they illustrate what a pre-trained general model can do without an additional task-specific dataset. The dataset and candidate sets limit what those results establish.

    Existing large datasets and efficient supervised models make CLIP less attractive for many standard on-demand surveillance tasks. It also does not directly handle common tasks such as object detection or semantic segmentation. Yet reducing the need for a custom dataset or specialized model could make niche surveillance applications easier to build. That change in accessibility deserves attention alongside the measured accuracy.

Conclusion

By bringing large-scale natural-language pre-training into computer vision, CLIP shows how image–text matching can support many tasks and zero-shot transfer. Let us collect its contributions and the work still ahead:

Key Contributions:

  1. Multimodal learning:

    CLIP learns from images and text together. Large-scale contrastive pre-training and natural-language prompts make one representation adaptable to different classification tasks.

  2. Zero-shot transfer:

    Prompts can define candidate classes without labeled training examples from the target task. The results are strong on many benchmarks, with the task and candidate wording still determining the outcome.

  3. Scalability:

    At sufficient scale, CLIP can approach some task-specific supervised baselines. The paper demonstrates a useful scaling trend, while its compute demands keep that trend from being a free solution.

Future Directions:

  1. Interpretability:

    Text descriptions can make a candidate score easier to name, but they do not explain every internal decision or provide a detailed causal account. Complex network representations and biased training data can make an apparently plausible description unreliable.

  2. Performance and few-shot learning:

    There is room to improve against specialized supervised systems and to learn efficiently from a few labeled examples without losing zero-shot strengths.

The authors also examine possible social impacts and acknowledge the people and software tools behind the work. CLIP opens a path for further multimodal research, while the limitations above remain part of its story.