BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2208.06366
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{visiontransformersbeitv22022,
title = {{BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers}},
author = {Zhiliang Peng and Li Dong and Hangbo Bao and Qixiang Ye and Furu Wei},
year = {2022},
eprint = {2208.06366},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2208.06366}
}
Do you remember BEiT?
If you've forgotten, let's quickly review, or you won't be able to understand this paper:
The earlier BEiT uses a separately pretrained image tokenizer to assign a discrete label to each patch. Its vocabulary has 8,192 possible codes; an image does not contain exactly 8,192 features. The upper part of the figure shows the tokenizer and image-reconstruction decoder. That dVAE was trained separately and remains frozen during BEiT's MIM pretraining. The lower part shows the masked image and the BEiT encoder being updated: it predicts codes at masked positions, much as BERT predicts masked words. Think of it as an “image dictionary,” although this earlier dictionary was mainly trained to reconstruct pixels.
Defining the Problem
Speaking of MIM, MAE is probably the best-known recent example. With a high mask ratio and pixel reconstruction, MAE showed that the task can learn transferable visual representations. It usually masks patches, rather than sampling individual pixels independently. The results were strong and inspired a great deal of follow-up work.
Still, a pixel-reconstruction objective emphasizes low-level details. Such a model can learn semantics, but the objective does not directly ask every masked position to match a high-level semantic feature. The earlier BEiT predicts discrete tokens, yet its tokenizer was mainly trained through image reconstruction. So what counts as “high-level” semantic information here? The authors want to compress a teacher model's semantic representations into a new discrete visual vocabulary and use those codes to supervise MIM. Patchwise prediction also may not push enough image-wide information into the [CLS] token, motivating a separate global-representation branch.
Solving the Problem
Model Architecture
Paper Figure 2 shows how the visual tokenizer is trained using vector-quantized knowledge distillation (VQ-KD). That is a sudden pile of terminology, so let us follow the picture. A ViT encoder produces patch features on the left. A codebook in the middle selects discrete codes. A decoder on the right tries to reconstruct semantic features supplied by the teacher. The red arrow marks the straight-through gradient across the quantization step.
-
Compute image features. Split into patches. A image with patches has positions. The ViT encoder processes the entire patch sequence and produces at each position; it does not run a separate, independent ViT for each patch.
-
Look up a code. Let contain vectors of dimension . Each selects the closest code after L2 normalization:
Euclidean distance between normalized vectors is equivalent to comparing their cosine directions. is a code index. Codebook vectors are 32-dimensional; the encoder and decoder operate in larger hidden spaces, with projections between them.
-
Distill from a teacher. The selected code vectors enter a Transformer decoder. Its output is aligned with a DINO or CLIP teacher feature for that patch. Maximizing teaches the discrete codes to retain information from the teacher. The semantics come from the teacher's learned representation, not from nowhere.
The quantization is not differentiable. The authors copy gradients from the decoder input back to the encoder using a straight-through estimator and use stop-gradient terms to align encoder output and codebook. Paper Equation (2) can be written as
The two distance terms pull the selected code toward the encoder feature and the encoder feature toward the selected code. The sg side receives no gradient from that term. The implementation also uses an exponential moving average (EMA) to update the codebook; the displayed objective alone is not a full account of its update rule. To reduce codebook collapse, the authors use normalized lookup and a 32-dimensional codebook projected back to a higher-dimensional decoder space. The hidden “code” is therefore a discrete index that compresses continuous teacher features into a target MIM can predict.
Pretraining Method
After training the tokenizer, BEiT v2's MIM training begins. On the left of paper Figure 3, approximately 40% of patches are selected through blockwise masking. Each masked patch receives the same learnable [MASK] embedding. A linear MIM head on the pretraining backbone's final patch outputs predicts tokenizer codes for the original unmasked image, only at masked positions:
Here is the pretraining image set and the masked positions. Codes come from the separately trained tokenizer. The Figure 2 quantization decoder is not needed for downstream inference or fine-tuning.
Global Representation Pretraining
The Patch Aggregation branch on the right of Figure 3 trains [CLS] in the MIM backbone, not inside the visual tokenizer. The final-layer [CLS] token is concatenated with patch tokens from intermediate layer . A shallow Transformer decoder and a parameter-shared MIM head again predict masked codes. The additional loss encourages the backbone to gather image-wide information into [CLS] rather than solve only local patch predictions:
The shallow aggregation decoder is used only during pretraining; downstream tasks retain the backbone representation. Linear probing uses [CLS] for models with aggregation, but averages patch tokens for the baseline without it. Keep that readout difference in mind when interpreting Table 5.
Model Configuration
The visual tokenizer uses a ViT-B/16 encoder and a three-layer Transformer decoder, with and code dimension . It is trained on ImageNet-1K images. The main experiments use CLIP-B/16 as teacher. CLIP itself was pretrained on large-scale image–text pairs, and that teacher contributes outside image–text supervision to the training chain. The paper also tests a DINO teacher trained only on ImageNet-1K.
MIM pretraining uses ImageNet-1K images at , a ViT-B/16 or ViT-L/16 backbone, and about 40% blockwise masking. The default aggregation head has two layers; for the B model and for the L model.
Discussion
Comparison with Other Methods
Paper Table 1 lists ImageNet-1K fine-tuning top-1 alongside ADE20K mIoU. Align backbone and pretraining duration first. With ViT-B/16 and 300 epochs, BEiT v2 reports 85.0%/52.7, against 84.4%/52.4 for MVP in that group. At 1,600 epochs, BEiT v2 B reaches 85.5%/53.1. ViT-L/16 reaches 86.6%/55.0 at 300 epochs and 87.3%/56.7 at 1,600. The additional 88.4%/57.5 row includes intermediate fine-tuning on ImageNet-21K, so it is not part of the ImageNet-1K-only comparison.
ADE20K uses UperNet, inputs, and 160K fine-tuning iterations. Classification top-1 and segmentation mIoU measure different tasks. The 80.1% linear probe in Table 2 is also distinct from the 85.5% fine-tuning in Table 1: a linear probe freezes the backbone and trains only a classifier, while fine-tuning updates the entire model. The reported comparisons support this combination of semantic tokenizer and aggregation under the specified settings, with differing teachers, data, and schedules across methods.
Ablation Study - Codebook Size
Paper Table 4 varies tokenizer encoder and decoder depth as well as code embedding dimension 16/32/64 with 8,192 codes fixed. Thus “size” can mean the number of codes or the dimension of each one. At the default three-layer decoder and , codebook usage is 95% and linear probe 77.9%. A six-layer decoder improves reconstruction loss from 0.145 to 0.136, while usage drops to 77%, linear probe to 63.0%, and fine-tuning from 84.7% to 84.6%.
Ablation Study - Patch Aggregation Strategy
In paper Table 5, the baseline without aggregation gives 84.7% fine-tuning/77.9% linear probe/51.9 mIoU. At with a two-layer head and shared MIM head, these rise to 85.0/80.1/52.7; linear probe improves by 2.2 percentage points. Both fine-tuning and linear probe are higher with this two-layer head than with one or three layers. A three-layer head gives 78.1% linear probe; one layer gives 78.9%, and separate MIM heads give 79.5%. ADE20K does not follow the same ranking by head depth.
Ablation Study - Teacher Objective Analysis
Paper Table 6 reports 84.4% ImageNet/49.2 ADE20K for BEiT v2 with a DINO teacher and 300 epochs; with CLIP, it reports 85.0%/52.7. The gray DINO 83.6/46.8 and CLIP 84.9 rows directly fine-tune teacher checkpoints; no direct CLIP ADE20K result is listed. Direct fine-tuning of the VQ-KD encoder with CLIP as its target reaches 83.6% on ImageNet, below CLIP's 84.9%; adding MIM pretraining reaches 85.0%. DINO and CLIP use different teacher-training data, so their difference cannot all be assigned to the distillation method.
Codebook Visualization
Finally, let us look at the codebook. Paper Figure 4 groups ImageNet validation patches by discrete code: includes eyes from humans, cats, dogs, fish, and snakes. The picture also shows tongue, nose, and ear examples. It is a useful glimpse of semantic similarity, but only selected codes are displayed. The other codes in the 8,192-entry codebook require separate inspection.
Conclusion
The codebook is an appealing idea.
Looking at the direction of this research, a codebook can help tokenize images and train a better visual tokenizer and MIM model. It also looks like a useful bridge to multimodal work. Discrete codes let a model draw semantic features from large amounts of unlabeled data, reducing its dependence on manual annotations. Perhaps domains with scarce labels can benefit from this kind of self-supervised learning. With a CLIP teacher, though, the teacher's earlier large-scale image–text training still belongs in the resource accounting.
Overall, VQ-KD improves BEiT's pretraining by taking MIM toward higher-level semantics. Patch aggregation helps the model collect global information and narrows the gap between the pretraining objective and downstream use.
Next, let us take the idea of discrete encoding into multimodal work with BEiT-3.