Training data-efficient image transformers & distillation through attention
- Publication
- ICML 2021Conference
- First submitted to arXiv
- arXiv:2012.12877
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{visiontransformersdeit2020,
title = {{Training data-efficient image transformers \& distillation through attention}},
author = {Hugo Touvron and Matthieu Cord and Matthijs Douze and Francisco Massa and Alexandre Sablayrolles and Hervé Jégou},
year = {2020},
eprint = {2012.12877},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2012.12877}
}
Earlier, we reviewed ViT.
That paper explored Transformers for images. With large datasets, ViT surpassed conventional convolutional networks in its experiments and showed considerable potential.
One conclusion from that paper was:
- With too little training data, ViT did not generalize well under the original recipe.
Problem Definition
The authors of this paper ask whether:
- Knowledge distillation can reduce the model's dependence on training data while improving its performance. Their ImageNet-only result also relies on a stronger training recipe; distillation is one part of the story.
Solution
Model Architecture
Don't be intimidated by the image.
This isn't something entirely new. If you remove the distillation part, you have the original ViT.
The left part of the image (CE loss) represents the standard ViT training process.
Everything we discuss here starts with the distillation token on the right.
Model Distillation
The authors add a token to the original ViT architecture and let it join the model's self-attention. At the output, they supervise this token through a Teacher-Student setup. The class head learns from human labels, while the distillation head learns from teacher predictions; the teacher's weights are not copied into the student.
The Teacher is a pre-trained model, which can be a ConvNet or another architecture.
During training, the Teacher provides the output classes for the Student to learn but does not participate in gradient updates.
The student is the Transformer being trained: primarily DeiT-B, corresponding to ViT-B/16, with Tiny and Small variants also evaluated.
There are two types of supervision:
-
Soft Distillation
Soft distillation minimizes the KL divergence between the teacher's and student's softmax outputs. Let and be their logits, the temperature, the weight balancing KL and cross-entropy against the ground-truth label , and the softmax function. The paper writes the usual, single-output objective as:
Here, is the loss against the true label, and KL is the distillation loss. This equation describes usual distillation with one student output . Figure 2 instead has separate class and distillation heads for those two sources of supervision. The paper prints the KL arguments in the order above; its released training code computes the teacher-to-student divergence.
During training, the weight of the soft distillation and cross-entropy loss is adjusted using .
-
Hard-Label Distillation
Hard distillation uses the teacher’s most likely class as a pseudo-label. It need not equal the human-annotated ground truth.
Let be the teacher's hard decision. Equation (3) again describes the usual, single-output objective:
Data augmentation can change the teacher's prediction. This hard-label choice needs no temperature parameter and is conceptually simpler: plays the same role as . In Figure 2's two-head model, however, the class head receives and the distillation head receives ; the two losses still have equal weight. The added token and head remain trainable.
Ground-truth labels can also be smoothed: the true class gets probability and the remaining is shared among the other classes. The paper fixes in experiments using true labels.
Teacher Network
For the teachers compared in Table 2, a RegNetY convolutional teacher gives the DeiT-B student better results than a DeiT-B teacher. The DeiT-B teacher scores 81.8% top-1 and its student reaches 81.9%; even the RegNetY-4GF teacher, at 80.0%, yields a student at 82.7%. RegNetY-16GF scores 82.9% as teacher and 83.1% as student.
The authors hypothesize that the student may inherit a convolutional inductive bias through distillation. This explains the observed pattern.
Discussion
Which Distillation Strategy Is Best?
The table above shows the performance of different distillation strategies.
- The upper rows use one output head and no distillation token. No distillation and usual soft distillation give the same score for DeiT-B at 224×224: 81.8%. Usual hard distillation raises it to 83.0%, a gain of 1.2 percentage points; the gains for the other model sizes differ.
- The lower rows train the two-head model with both the label and the teacher prediction. For DeiT-B at 224×224, the class head, distillation head and their late fusion score 83.0%, 83.1% and 83.4%, respectively.
At inference, only the student runs. The paper compares each head with late fusion of their softmax outputs; it does not simply average the two embedding vectors. Fusion scores highest for DeiT-S, DeiT-B and DeiT-B at 384 resolution. DeiT-Ti is the exception: its distillation head alone scores 74.6%, just above fusion at 74.5%.
ImageNet Experiment Results
In the literature on image classification methods, authors typically compare accuracy trade-offs with other standards (e.g., FLOPs, number of parameters, network size).
Figure 1 plots top-1 accuracy against throughput, rather than FLOPs or single-image latency. Undistilled DeiT trails EfficientNet slightly, showing how much of the gap between ViT and CNNs the ImageNet-only recipe closes. Some of the RegNetY-distilled DeiT points surpass EfficientNet on both axes. Table 5 gives the exact comparison with the ImageNet-1k-only ViT-B/16 at 77.9%: undistilled DeiT-B at 384 reaches 83.1%, a gain of 5.2 percentage points; the 1000-epoch distilled DeiT-B at 224 reaches 84.2%, a gain of 6.3 points. These models differ in training setup, so neither gain is solely due to the distillation token.
Table 5 reports 81.8%/83.1% top-1 for undistilled DeiT-B at 224/384 resolution. The distilled model trained for 300 epochs reaches 83.4%/84.5%; with 1000 epochs, it reaches 84.2%/85.2%. The latter exceeds the 84.15% of a JFT-300M-pretrained ViT-B evaluated at 384 by 1.05 points, although the data and training costs differ. For the 85.2% model, ImageNet-ReaL/V2 scores are 89.3%/75.2%.
The table also evaluates ImageNet-V2, which contains newly collected test images, and ImageNet-ReaL, which relabels the original validation images. They help check whether the findings depend too much on the original validation labels and images. Throughput is measured on one 16 GB V100 with the largest batch that fits each model. On that measure, DeiT-B is competitive with the listed models, though some have higher accuracy or faster throughput.
Conclusion
DeiT reaches 83.1% top-1 without external data: it trains on ImageNet alone on one 8-GPU node. That result comes from the training recipe, including strong augmentation and regularization; distillation further improves it. The reference model takes about 53 hours of pretraining and about 20 more hours of fine-tuning at 384, roughly three days in total. The best 85.2% result uses a 1000-epoch distilled run plus fine-tuning, so it has a different training cost.
More importantly, the distillation token lets the student learn from the teacher through attention while its class token learns from the human label. In the paper's ImageNet accuracy-versus-throughput comparison, the distilled models are competitive with convolutional networks.
For us, this is an encouraging way to train a high-performing ViT without a JFT-scale pretraining set. It still takes several GPUs and a substantial amount of compute.