Back to paper library
arXiv 2022Vision TransformersPaper notes

BEiT v2

Hidden Codes in High Dimensions

Original paper

BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

Paper authors

Zhiliang Peng · Li Dong · Hangbo Bao · Qixiang Ye · Furu Wei

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2208.06366
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{visiontransformersbeitv22022,
  title = {{BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers}},
  author = {Zhiliang Peng and Li Dong and Hangbo Bao and Qixiang Ye and Furu Wei},
  year = {2022},
  eprint = {2208.06366},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.06366}
}
On this page

Do you remember BEiT?

If you've forgotten, let's quickly review, or you won't be able to understand this paper:

Original BEiT paper Figure 1; Bao et al., arXiv:2106.08254; a recap, not the new BEiT v2 architecture

The earlier BEiT uses a separately pretrained image tokenizer to assign a discrete label to each patch. Its vocabulary has 8,192 possible codes; an image does not contain exactly 8,192 features. The upper part of the figure shows the tokenizer and image-reconstruction decoder. That dVAE was trained separately and remains frozen during BEiT's MIM pretraining. The lower part shows the masked image and the BEiT encoder being updated: it predicts codes at masked positions, much as BERT predicts masked words. Think of it as an “image dictionary,” although this earlier dictionary was mainly trained to reconstruct pixels.

Defining the Problem

Speaking of MIM, MAE is probably the best-known recent example. With a high mask ratio and pixel reconstruction, MAE showed that the task can learn transferable visual representations. It usually masks patches, rather than sampling individual pixels independently. The results were strong and inspired a great deal of follow-up work.

Still, a pixel-reconstruction objective emphasizes low-level details. Such a model can learn semantics, but the objective does not directly ask every masked position to match a high-level semantic feature. The earlier BEiT predicts discrete tokens, yet its tokenizer was mainly trained through image reconstruction. So what counts as “high-level” semantic information here? The authors want to compress a teacher model's semantic representations into a new discrete visual vocabulary and use those codes to supervise MIM. Patchwise prediction also may not push enough image-wide information into the [CLS] token, motivating a separate global-representation branch.

Solving the Problem

Model Architecture

Paper Figure 2 shows how the visual tokenizer is trained using vector-quantized knowledge distillation (VQ-KD). That is a sudden pile of terminology, so let us follow the picture. A ViT encoder produces patch features on the left. A codebook in the middle selects discrete codes. A decoder on the right tries to reconstruct semantic features supplied by the teacher. The red arrow marks the straight-through gradient across the quantization step.

BEiT v2 paper Figure 2; Peng et al., arXiv:2208.06366v2
  1. Compute image features. Split x∈RH×W×Cx\in\mathbb{R}^{H\times W\times C} into N=HW/P2N=HW/P^2 patches. A 224×224224\times224 image with 16×1616\times16 patches has 14×14=19614\times14=196 positions. The ViT encoder processes the entire patch sequence and produces hih_i at each position; it does not run a separate, independent ViT for each patch.

  2. Look up a code. Let V∈RK×DV\in\mathbb{R}^{K\times D} contain KK vectors of dimension DD. Each hih_i selects the closest code after L2 normalization:

    zi=arg⁡min⁡j∥ℓ2(hi)−ℓ2(vj)∥22.z_i=\arg\min_j\left\|\ell_2(h_i)-\ell_2(v_j)\right\|_2^2.

    Euclidean distance between normalized vectors is equivalent to comparing their cosine directions. ziz_i is a code index. Codebook vectors are 32-dimensional; the encoder and decoder operate in larger hidden spaces, with projections between them.

  3. Distill from a teacher. The selected code vectors enter a Transformer decoder. Its output oio_i is aligned with a DINO or CLIP teacher feature tit_i for that patch. Maximizing cos⁡(oi,ti)\cos(o_i,t_i) teaches the discrete codes to retain information from the teacher. The semantics come from the teacher's learned representation, not from nowhere.

The quantization arg⁡min⁡\arg\min is not differentiable. The authors copy gradients from the decoder input back to the encoder using a straight-through estimator and use stop-gradient terms to align encoder output and codebook. Paper Equation (2) can be written as

max⁡∑x∈D∑i=1N[cos⁡(oi,ti)−∥sg⁡[ℓ2(hi)]−ℓ2(vzi)∥22−∥ℓ2(hi)−sg⁡[ℓ2(vzi)]∥22].\max\sum_{x\in D}\sum_{i=1}^{N}\left[\cos(o_i,t_i)-\|\operatorname{sg}[\ell_2(h_i)]-\ell_2(v_{z_i})\|_2^2-\|\ell_2(h_i)-\operatorname{sg}[\ell_2(v_{z_i})]\|_2^2\right].

The two distance terms pull the selected code toward the encoder feature and the encoder feature toward the selected code. The sg side receives no gradient from that term. The implementation also uses an exponential moving average (EMA) to update the codebook; the displayed objective alone is not a full account of its update rule. To reduce codebook collapse, the authors use normalized lookup and a 32-dimensional codebook projected back to a higher-dimensional decoder space. The hidden “code” is therefore a discrete index that compresses continuous teacher features into a target MIM can predict.

Pretraining Method

After training the tokenizer, BEiT v2's MIM training begins. On the left of paper Figure 3, approximately 40% of patches are selected through blockwise masking. Each masked patch receives the same learnable [MASK] embedding. A linear MIM head on the pretraining backbone's final patch outputs predicts tokenizer codes for the original unmasked image, only at masked positions:

LMIM=−∑x∈D∑i∈Mlog⁡p(zi∣xM).L_{\mathrm{MIM}}=-\sum_{x\in D}\sum_{i\in M}\log p(z_i\mid x^M).
BEiT v2 paper Figure 3; Peng et al., arXiv:2208.06366v2

Here DD is the pretraining image set and MM the masked positions. Codes ziz_i come from the separately trained tokenizer. The Figure 2 quantization decoder is not needed for downstream inference or fine-tuning.

Global Representation Pretraining

The Patch Aggregation branch on the right of Figure 3 trains [CLS] in the MIM backbone, not inside the visual tokenizer. The final-layer [CLS] token is concatenated with patch tokens from intermediate layer ll. A shallow Transformer decoder and a parameter-shared MIM head again predict masked codes. The additional loss LMIMcL^c_{\mathrm{MIM}} encourages the backbone to gather image-wide information into [CLS] rather than solve only local patch predictions:

Ltotal=LMIM+LMIMc.L_{\mathrm{total}}=L_{\mathrm{MIM}}+L^c_{\mathrm{MIM}}.

The shallow aggregation decoder is used only during pretraining; downstream tasks retain the backbone representation. Linear probing uses [CLS] for models with aggregation, but averages patch tokens for the baseline without it. Keep that readout difference in mind when interpreting Table 5.

Model Configuration

The visual tokenizer uses a ViT-B/16 encoder and a three-layer Transformer decoder, with K=8192K=8192 and code dimension D=32D=32. It is trained on ImageNet-1K images. The main experiments use CLIP-B/16 as teacher. CLIP itself was pretrained on large-scale image–text pairs, and that teacher contributes outside image–text supervision to the training chain. The paper also tests a DINO teacher trained only on ImageNet-1K.

MIM pretraining uses ImageNet-1K images at 224×224224\times224, a ViT-B/16 or ViT-L/16 backbone, and about 40% blockwise masking. The default aggregation head has two layers; l=9l=9 for the B model and l=21l=21 for the L model.

Discussion

Comparison with Other Methods

Paper Table 1 lists ImageNet-1K fine-tuning top-1 alongside ADE20K mIoU. Align backbone and pretraining duration first. With ViT-B/16 and 300 epochs, BEiT v2 reports 85.0%/52.7, against 84.4%/52.4 for MVP in that group. At 1,600 epochs, BEiT v2 B reaches 85.5%/53.1. ViT-L/16 reaches 86.6%/55.0 at 300 epochs and 87.3%/56.7 at 1,600. The additional 88.4%/57.5 row includes intermediate fine-tuning on ImageNet-21K, so it is not part of the ImageNet-1K-only comparison.

BEiT v2 paper Table 1; Peng et al., arXiv:2208.06366v2; grouped by model size, epochs, and ImageNet-21K use

ADE20K uses UperNet, 512×512512\times512 inputs, and 160K fine-tuning iterations. Classification top-1 and segmentation mIoU measure different tasks. The 80.1% linear probe in Table 2 is also distinct from the 85.5% fine-tuning in Table 1: a linear probe freezes the backbone and trains only a classifier, while fine-tuning updates the entire model. The reported comparisons support this combination of semantic tokenizer and aggregation under the specified settings, with differing teachers, data, and schedules across methods.

Ablation Study - Codebook Size

Paper Table 4 varies tokenizer encoder and decoder depth as well as code embedding dimension 16/32/64 with 8,192 codes fixed. Thus “size” can mean the number of codes or the dimension of each one. At the default three-layer decoder and 8192×328192\times32, codebook usage is 95% and linear probe 77.9%. A six-layer decoder improves reconstruction loss from 0.145 to 0.136, while usage drops to 77%, linear probe to 63.0%, and fine-tuning from 84.7% to 84.6%.

BEiT v2 paper Table 4; Peng et al., arXiv:2208.06366v2

Ablation Study - Patch Aggregation Strategy

In paper Table 5, the baseline without aggregation gives 84.7% fine-tuning/77.9% linear probe/51.9 mIoU. At l=9l=9 with a two-layer head and shared MIM head, these rise to 85.0/80.1/52.7; linear probe improves by 2.2 percentage points. Both fine-tuning and linear probe are higher with this two-layer head than with one or three layers. A three-layer head gives 78.1% linear probe; one layer gives 78.9%, and separate MIM heads give 79.5%. ADE20K does not follow the same ranking by head depth.

BEiT v2 paper Table 5; Peng et al., arXiv:2208.06366v2

Ablation Study - Teacher Objective Analysis

Paper Table 6 reports 84.4% ImageNet/49.2 ADE20K for BEiT v2 with a DINO teacher and 300 epochs; with CLIP, it reports 85.0%/52.7. The gray DINO 83.6/46.8 and CLIP 84.9 rows directly fine-tune teacher checkpoints; no direct CLIP ADE20K result is listed. Direct fine-tuning of the VQ-KD encoder with CLIP as its target reaches 83.6% on ImageNet, below CLIP's 84.9%; adding MIM pretraining reaches 85.0%. DINO and CLIP use different teacher-training data, so their difference cannot all be assigned to the distillation method.

BEiT v2 paper Table 6; Peng et al., arXiv:2208.06366v2

Codebook Visualization

Finally, let us look at the codebook. Paper Figure 4 groups ImageNet validation patches by discrete code: v7856v_{7856} includes eyes from humans, cats, dogs, fish, and snakes. The picture also shows tongue, nose, and ear examples. It is a useful glimpse of semantic similarity, but only selected codes are displayed. The other codes in the 8,192-entry codebook require separate inspection.

BEiT v2 paper Figure 4; Peng et al., arXiv:2208.06366v2

Conclusion

The codebook is an appealing idea.

Looking at the direction of this research, a codebook can help tokenize images and train a better visual tokenizer and MIM model. It also looks like a useful bridge to multimodal work. Discrete codes let a model draw semantic features from large amounts of unlabeled data, reducing its dependence on manual annotations. Perhaps domains with scarce labels can benefit from this kind of self-supervised learning. With a CLIP teacher, though, the teacher's earlier large-scale image–text training still belongs in the resource accounting.

Overall, VQ-KD improves BEiT's pretraining by taking MIM toward higher-level semantics. Patch aggregation helps the model collect global information and narrows the gap between the pretraining objective and downstream use.

Next, let us take the idea of discrete encoding into multimodal work with BEiT-3.