Back to paper library
arXiv 2021ICLR 2022Vision TransformersPaper notes

BEiT

Discrete Encoding Pre-Training

Original paper

BEiT: BERT Pre-Training of Image Transformers

Paper authors

Hangbo Bao · Li Dong · Songhao Piao · Furu Wei

Publication
ICLR 2022Conference
First submitted to arXiv
arXiv:2106.08254
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{visiontransformersbeit2021,
  title = {{BEiT: BERT Pre-Training of Image Transformers}},
  author = {Hangbo Bao and Li Dong and Songhao Piao and Furu Wei},
  year = {2021},
  eprint = {2106.08254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.08254}
}
On this page

The main comparisons in this paper are made between ViT and DeiT.

The primary objective is to improve the classification performance of the ViT architecture on ImageNet.

Problem Definition

BERT has achieved impressive results through MLM pre-training in the past, proving to be one of the most effective pre-training methods for denoising autoencoder-style models.

But BERT's idea has not worked as neatly for images. ViT tried masking input image patches and predicting targets such as quantized mean RGB, but its self-supervised result fell short of supervised pre-training. It masked the image input, not the model's parameters.

That unfinished problem was bound to get another try. Here the authors propose a new pre-training method:

  • Instead of directly predicting image pixel values, the authors suggest predicting "discrete labels" of images. This approach enables the model to learn higher-level features rather than just pixel values.

Problem Solving

Model Architecture

There are two architectures present here, which we will examine in sequence.

The upper branch is a discrete VAE (dVAE) image tokenizer trained for DALL-E. Its weights stay frozen during BEiT pre-training; it reads the original, unmasked image to produce target tokens. The decoder drawn beside it is not used during this stage.

Think of it as a trained image dictionary with 8192 visual-token IDs. They are entries learned during discrete encoding. The hope is that predicting these tokens teaches BEiT more than predicting raw pixels.

For details on the training method and more information, please refer to the original paper:


The second architecture, shown in the lower part of the image, is the model that actually participates in gradient updates.

This part is identical to the ViT we are familiar with. The difference is that instead of using image labels for supervision, BEiT introduces BERT's MLM mechanism:

  1. Patchify: The image is split into small patches, similar to ViT.

  2. Blockwise Masking: This part differs, as some patches are randomly masked. The masking logic is detailed in the paper:

    The masking logic uses "block-wise" rather than "token-level" masking like BERT. Each iteration first samples a block size, then an aspect ratio, and finally covers the resulting block.

    This process repeats until 40% of the image blocks are masked.

  3. Transformer Encoder: Selected patch embeddings are replaced by a shared, learned mask embedding. All positions enter the encoder; the original masked pixels are not supplied directly.

  4. Masked Patch Prediction: This part predicts the labels of the masked blocks, one of the 8192 labels mentioned earlier.

Training Method

BEiT's network architecture follows ViT-Base's design to ensure fair comparison. BEiT employs a 12-layer Transformer structure with a hidden size of 768, 12 attention heads, and a 3072 intermediate size in the FFN. The input patch size is 16 × 16, with a vocabulary size of 8192 visual tokens generated by a trained image tokenizer.

BEiT is pre-trained on the ImageNet-1K training set, containing about 1.2 million images. The pre-training process includes image augmentations like random resizing, cropping, horizontal flipping, and color jittering. Importantly, labels are not used during pre-training.

A 224×224 input is divided into a 14×14 grid of 16×16-pixel patches, totaling 196 tokens. Masking at most 75 gives about 38.3%, described in the paper as roughly 40%.

Pre-training lasted for approximately 500,000 steps (around 800 epochs) with a batch size of 2000. They used the Adam optimizer with β1=0.9\beta_1 = 0.9 and β2=0.999\beta_2 = 0.999, a learning rate set at 1.5×10−31.5 \times 10^{-3}, with the first 10 epochs as a warm-up period, followed by cosine learning rate decay. Weight decay was set at 0.05, and stochastic depth with a rate of 0.1 was employed.

Training utilized 16 Nvidia Tesla V100 32GB GPU cards, taking about five days.

Initialization starts with small weights. The official code uses a truncated normal with standard deviation 0.02 for linear weights and LayerNorm scale/bias of 1/0; not every parameter is uniformly sampled from [-0.02,0.02]. The output-projection and final FFN weights in layer ll are then multiplied by 1/2l1/\sqrt{2l}, not set to that constant.

Fine-tuning for Downstream Tasks

After pre-training BEiT, the authors added task-specific layers on the Transformer and fine-tuned the parameters on downstream tasks, similar to BERT.

The paper uses image classification and semantic segmentation as examples. For other visual tasks, the pre-train and fine-tune paradigm can be directly applied.

  • Image Classification: For image classification, the authors used a simple linear classifier as the task layer. The representations were aggregated using average pooling and passed to a softmax classifier.

  • Semantic Segmentation: For semantic segmentation, the authors followed the task layer used in SETR-PUP. They employed the pre-trained BEiT as the backbone encoder and combined it with several deconvolution layers as the decoder to generate segmentation. The model was fine-tuned end-to-end, similar to image classification.

Discussion

Performance on ImageNet

The table mixes Base and Large models and several pre-training conditions, so it needs careful reading. iGPT has 1.36B parameters. The ViT-JFT300M row uses self-supervised masked-patch prediction on Google's internal 300-million-image dataset, not supervised JFT training; its 79.9% result also has a very different pre-training data budget from ImageNet-1K methods.

BEiT-B scores 83.2% at 224×224, the same as MoCo v3-B in this table. BEiT-L reaches 85.2%, above MoCo v3-L's 84.1%. Thus BEiT does well here, though it does not beat every listed method at every size.

To further validate BEiT's performance, the authors fine-tuned it on a 224 × 224 resolution and then additionally fine-tuned on images with a 384 × 384 resolution. The results indicate that the higher resolution significantly improves BEiT's performance on ImageNet, even surpassing ViT384\text{ViT}_{384}, which was pre-trained on ImageNet-22K.

The authors also scale BEiT from Base to Large. At 384×384, that raises Top-1 by 1.7 percentage points, compared with 1.2 points for the ImageNet-22K supervised ViT pair. That suggests potential for larger models; this table alone does not test what happens when labeled data are scarce.

Ablation Studies

Finally, the authors ablate several components, using a 300-epoch pre-trained BEiT as the baseline. Replacing Blockwise masking with random masked positions lowers ImageNet accuracy from 82.86% to 82.77% and ADE20K segmentation from 44.65 to 42.93 mIoU. The segmentation change is larger.

Next, they remove visual-token prediction and instead predict the masked pixels, turning pre-training into pixel regression. ImageNet falls to 81.04% and segmentation to 41.38 mIoU. In this setup, visual-token prediction works better than simple pixel-level autoencoding.

The 81.04% classification result is also below the 81.8% of DeiT-B trained from scratch in the previous table. That table has no corresponding segmentation baseline for this claim. The result concerns this pixel-regression ablation, not every way to reconstruct pixels.

Removing both visual tokens and Blockwise masking lowers classification again, from 81.04% to 80.50%, and segmentation from 41.38 to 37.09 mIoU. In these pixel-regression experiments, Blockwise masking matters even more; the authors suggest it reduces the short-distance dependency problem.

Predicting all visual tokens instead of just masked ones lowers the scores to 82.59% and 40.93 mIoU. Removing masking as well while still predicting all visual tokens lowers them further to 81.67% and 36.73 mIoU. Extending pre-training from 300 to 800 epochs raises the baseline scores to 83.19% and 45.58 mIoU.

These ablation studies indicate that Blockwise masking and visual token prediction are essential to BEiT's success.

Conclusion

BEiT continues the unfinished self-supervised pre-training of ViT, successfully transferring BERT's success to the ViT architecture. By introducing a self-supervised pre-training framework, BEiT achieves excellent fine-tuning results on downstream tasks like image classification and semantic segmentation.

There is also an interesting side effect: without classification or segmentation labels during BEiT's masked pre-training, some attention maps already pick out semantic regions in the image above. They are not reliable segmentation masks, and downstream fine-tuning still uses labels. The reused DALL-E tokenizer was trained on external data as well. Even so, I can see why this result hints at later multimodal pre-training work.

Subsequent versions, BEiT-V2 and BEiT-V3, have further improvements, which we will discuss when they become relevant.