Back to paper library
arXiv 2020ICLR 2021Vision TransformersPaper notes

ViT

Pioneering a New World

Original paper

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Paper authors

Alexey Dosovitskiy · Lucas Beyer · Alexander Kolesnikov …

Show all 12 authors

Alexey Dosovitskiy · Lucas Beyer · Alexander Kolesnikov · Dirk Weissenborn · Xiaohua Zhai · Thomas Unterthiner · Mostafa Dehghani · Matthias Minderer · Georg Heigold · Sylvain Gelly · Jakob Uszkoreit · Neil Houlsby

Publication
ICLR 2021Conference
First submitted to arXiv
arXiv:2010.11929
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{visiontransformersvit2020,
  title = {{An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale}},
  author = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
  year = {2020},
  eprint = {2010.11929},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.11929}
}
On this page

In 2017, the introduction of the Transformer model caused a massive wave in the field of natural language processing (NLP), quickly becoming a dominant force.

Three years later, this wave has finally reached the realm of computer vision.

Problem Definition

In recent years, many studies have attempted to integrate attention mechanisms with convolutional networks.

Or they have replaced certain parts while keeping the convolutional network's structure unchanged.

These attempts show that image representation and training design matter.

However, the authors of this paper argue:

  • That's because you're doing it wrong!

We can completely abandon convolutions and directly use Transformers to process images.

The analogy is to treat each 16×16-pixel patch as a token—not to use a fixed 16×16 grid of tokens or convert image content into literal words.

Solution

Model Architecture

We are all familiar with the Transformer architecture.

The original translation Transformer has an encoder and decoder. ViT classification uses only the Transformer encoder followed by a classification head; no decoder is required.

When applying this structure to images, the first question to consider is:

  • How do you convert images into a sequence of words?

Here, the authors propose a method: patch it up!

Patchify

Given an image with dimensions 224 x 224, how would you divide it into smaller regions?

Manually cut it? Of course not!

The paper flattens patches and applies a shared linear projection. The Conv2d below implements the same operation with kernel size equal to stride.

Let's implement it:

python
import torch
import torch.nn as nn

# Assume image size is 224 x 224
dummy_img = torch.randn(1, 3, 224, 224)

# Patch size
patch_size = 16

# Embedding dimension
embed_dim = 768

# Patchify
emb_layer = nn.Conv2d(3, embed_dim, kernel_size=patch_size, stride=patch_size)

# After patching:
# input.shape = (1, 3, 224, 224)
# tokens.shape = (1, 768, 14, 14)
tokens = emb_layer(dummy_img)

Here, we set the patch size to 16 x 16 and the embedding dimension to 768.

Using the stride in the convolution, we can slide across the image in non-overlapping windows, dividing the 224 x 224 image into 14 x 14 patches.

In the original Transformer, each token in the text sequence is embedded. Here, we do the same: each image patch is embedded, meaning each 16 x 16 x 3 region is linearly transformed into a 768-dimensional vector.

Finally, we flatten these patches into a sequence:

python
tokens = tokens.flatten(2) # (1, 768, 14, 14) -> (1, 768, 196)
tokens = tokens.permute(2, 0, 1) # (1, 768, 196) -> (196, 1, 768)

This example uses PyTorch’s batch_first=False layout (sequence, batch, embedding). With batch_first=True, the order is (batch, sequence, embedding); this is an interface choice, not a universal Transformer requirement.

The example produces 196 patch embeddings. A complete ViT adds a learned class token, making 197 tokens, and position embeddings. After the encoder, the class-token representation feeds the classification head.

And Then?

That's it.

From here, you can treat it like any other NLP Transformer.

Wait, not so fast!

Below is the parameter setting for ViT:

Lack of Inductive Bias

ViT has fewer image-specific inductive biases than CNNs, not none. Patch extraction, shared projection and position embeddings still impose structure.

The MLP applies the same mapping to each token, while self-attention connects all tokens. The model imposes less 2D locality than a CNN; the complete ViT is not translation-invariant.

Therefore, the model must learn from scratch and understand: what is an image? What are the features of an image?

This helps explain the paper’s emphasis on data scale, but it does not by itself explain the research timeline or imply that all earlier work was abandoned.

Training Data Must Be Large

In this pretraining experiment, larger ViTs lag behind some ResNet (BiT) baselines on ImageNet linear 5-shot evaluation when the JFT subset is small; ViT-L/16 improves as the subset grows.

The two gray lines are ResNet50x1 and ResNet152x2 (BiT); the colored lines are ViTs. The horizontal axis is the number of JFT pretraining examples. At about 100M examples, ViT-L/16 already exceeds both ResNet curves shown; it does not need 300M to do so. These observations concern the paper’s recipe, not every later ViT training method.

The authors believe:

  • For small datasets, the inductive bias of convolutional networks is crucial.
  • For large datasets, learning relevant patterns directly from the data is sufficient!

Can Be Even Larger

What happens if we increase pretraining compute?

In this experiment, the authors used three model families, plotting total JFT-300M pretraining compute on the horizontal axis:

  • ViT
  • ResNet
  • Hybrid Model

With the pretraining dataset fixed at JFT-300M, ViT usually achieves better transfer accuracy than ResNet at similar compute.

Moreover, the Hybrid Model performs slightly better than ViT at lower compute, but this difference disappears at higher compute.

Finally, ViT shows no sign of saturation within the tested range, indicating more potential to be explored.

Discussion

What Does ViT See?

The left panel shows 28 principal components of learned ViT-L/32 patch-projection filters, not a self-attention map.

  • Patch-projection filters (left)

    The principal components show color and spatial patterns within a patch, illustrating the learned projection basis. Attention distance is analyzed in the right panel.

  • Position Embeddings (center)

    Patches that are spatially close have similar embeddings, indicating that these embeddings can encode the distance relationships between patches within the image.

    The embeddings show row-column structures, sometimes revealing significant sinusoidal structures for larger grids.

  • Attention Distance (right)

    This "attention distance" is akin to the receptive field size in CNNs.

    The average attention distance varies greatly across different heads in lower layers; some heads focus on most of the image, while others concentrate on the query position or nearby small regions.

    Overall, attention distance tends to grow with depth. In the network's latter half, most heads have long attention distances, indicating a focus on global information in these layers.

Predict with [CLS] or GAP?

In this paper, the authors used two different methods for classification tasks:

  • [CLS] Prediction

    This is common in NLP, using the first token of the sequence to represent the entire sequence.

    This approach also works well in the image domain.

  • GAP Prediction

    Here, GAP (Global Average Pooling) averages the output embeddings of the image patch tokens to obtain a classification vector.

    Initially, the authors used this method but found it performed very poorly!

    Detailed analysis revealed the issue wasn't GAP itself: reusing the class-token learning rate of 8×10−48 \times 10^{-4} was too high. Lowering it to 3×10−43 \times 10^{-4} brought the results close together.

After adjustments, both prediction methods achieved good results.

Self-Supervised Training

Inspired by MLM, the authors selected 50% of patch embeddings as prediction targets. Of those selected, 80% were replaced with a learnable [MASK] embedding, 10% with another patch embedding, and 10% were left unchanged.

Next, they trained on the JFT dataset, running the model for 1 million steps (equivalent to 14 epochs), with a batch size of 4096. During training, they used the Adam optimizer, set the base learning rate to 2×10−42 \times 10^{-4}, warmed up for the first 10,000 steps, and adopted a cosine learning rate decay strategy.

In this process, the authors tried different prediction targets, including:

  1. Predicting mean patch RGB with 3 bits per channel, giving 512 color classes
  2. Predicting the color at each of the 16 positions in a 4×4 downsampled patch, with 512 possible colors per position
  3. Regressing all RGB pixels of a 16×16 patch with L2 loss

Experiments found that all these methods achieved good results (L2 slightly inferior).

Finally, the authors chose the best-performing first method because it performed best in few-shot learning. Additionally, they also tried a 15% masking rate but found that this setting performed slightly worse.

The reported self-supervised ViT-B/16 reaches 79.9% ImageNet accuracy: about 2 percentage points above training from scratch and 4 points below supervised pretraining. This is not a fixed gap for all masked-patch methods.

Other Considerations

Besides the main content, here are some training tips and considerations:

  1. JFT pretraining uses weight decay 0.1. Other pretraining datasets use different settings; fine-tuning uses no weight decay.
  2. When the input image resolution changes, corresponding changes in the input sequence length (due to fixed patch size) necessitate interpolation of learned position embeddings on their two-dimensional grid.
  3. The paper pretrains all models with Adam and fine-tunes with momentum SGD (momentum 0.9). Its appendix compares transfer from two ResNets pretrained on JFT with Adam or SGD; Adam has a slightly better average there.
  4. In the ViT-B/16 ImageNet 5-shot ablation, 1-D learnable, 2-D learnable, and relative position encodings yield similar results; omitting position information performs worse.

Conclusion

This paper explores the application of Transformers to the image domain, proposing a novel architecture, ViT.

ViT outperforms traditional convolutional networks on large datasets and demonstrates greater potential in experiments.

The publication of this paper marks the successful application of the Transformer architecture in the image domain, opening new directions for future research.