Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
- Publication
- ICCV 2021Conference
- First submitted to arXiv
- arXiv:2102.12122
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{visiontransformerspvt2021,
title = {{Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions}},
author = {Wenhai Wang and Enze Xie and Xiang Li and Deng-Ping Fan and Kaitao Song and Ding Liang and Tong Lu and Ping Luo and Ling Shao},
year = {2021},
eprint = {2102.12122},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2102.12122}
}
ViT has achieved remarkable results in image classification.
It officially marked the Transformer architecture's advance into computer vision.
Defining the Problem
ViT linearly projects flattened 16×16-pixel patches, equivalently implemented by Conv2d with kernel and stride 16. A 224×224 image becomes a 14×14 token grid, not a 14×14-pixel image. This coarse output grid makes direct dense localization harder.
- A coarse output grid limits spatial localization; information within each patch depends on its feature representation.
Obviously, we need more refined features for dense prediction tasks. So, what if we change the 16 x 16 convolution to a 2 x 2 convolution?
- Of course not!
Let's take a 224 x 224 image as an example:
- Using a 16 x 16 patch size, we get 14 x 14, totaling 196 input tokens.
- Using a 2 x 2 patch size, we get 112 x 112, totaling 12,544 input tokens.
Imagine calculating a 12,544 x 12,544 self-attention matrix...
Clearly, no one would consider this a good idea.
Solving the Problem
Model Architecture
The above image shows the design of the PVT architecture. While it may look complex, it essentially follows the same principles as convolutional neural network (CNN) architectures.
Hierarchical Structure
PVT has four stages with output strides 4, 8, 16 and 32. The first uses 4×4 patches; subsequent stage transitions halve each spatial dimension.
Each transition groups non-overlapping patches and linearly projects them, equivalently implemented by Conv2d with matching kernel and stride. “Pure Transformer” describes the attention-based backbone, not a ban on Conv2d as an implementation primitive. A 3×224×224 input projected with kernel/stride 4 to 64 channels gives 64×56×56 features.
So, in the first stage, the input to the Transformer encoder is:
- Sequence Length: 3136 (all image patches, i.e., 56 x 56)
- Feature Number: 64 (features per patch, i.e., the number of channels)
Spatial Reduction Attention (SRA)
After obtaining the feature maps, we realize that a (56 x 56) x (56 x 56) self-attention matrix is still enormous and needs serious handling.
The authors propose "Spatial Reduction Attention" (SRA): keep the query (Q) token count, reduce the key (K) and value (V) token counts spatially, then project them back to the stage's channel dimension. For example, an attention map of (56 x 56) x (56 x 56) becomes 3136 x 49 if K and V are reduced to (7 x 7).
The attention matrix then has 1/64 as many elements. This lowers attention computation and memory use, although whole-model cost does not fall by the same factor.
As usual, let's look at the implementation. The highlighted code uses sr_ratio=[8,4,2,1] across the four stages; the last stage does not reduce K/V. In the example above, the first stage goes from 56 x 56 to 7 x 7. K/V token count falls by about , while Q and output token counts stay unchanged. It requires dim % num_heads == 0; the reduction branch also requires N == H * W.
# Reference:
# - https://github.com/whai362/PVT/blob/v2/classification/pvt.py
import torch
import torch.nn as nn
class SpatialReductionAttention(nn.Module):
def __init__(
self,
dim,
num_heads=8,
qkv_bias=False,
qk_scale=None,
attn_drop=0.,
proj_drop=0.,
sr_ratio=1
):
super().__init__()
self.dim = dim
self.num_heads = num_heads
head_dim = dim // num_heads
self.scale = qk_scale or head_dim ** -0.5
self.q = nn.Linear(dim, dim, bias=qkv_bias)
self.kv = nn.Linear(dim, dim * 2, bias=qkv_bias)
self.attn_drop = nn.Dropout(attn_drop)
self.proj = nn.Linear(dim, dim)
self.proj_drop = nn.Dropout(proj_drop)
self.sr_ratio = sr_ratio
if sr_ratio > 1:
self.sr = nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio)
self.norm = nn.LayerNorm(dim)
def forward(self, x, H, W):
B, N, C = x.shape
q = self.q(x).reshape(B, N, self.num_heads, C // self.num_heads).permute(0, 2, 1, 3)
if self.sr_ratio > 1:
x_ = x.permute(0, 2, 1).reshape(B, C, H, W)
x_ = self.sr(x_).reshape(B, C, -1).permute(0, 2, 1)
x_ = self.norm(x_)
kv = self.kv(x_).reshape(B, -1, 2, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
else:
kv = self.kv(x).reshape(B, -1, 2, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
k, v = kv[0], kv[1]
attn = (q @ k.transpose(-2, -1)) * self.scale
attn = attn.softmax(dim=-1)
attn = self.attn_drop(attn)
x = (attn @ v).transpose(1, 2).reshape(B, N, C)
x = self.proj(x)
x = self.proj_drop(x)
return x
Model Configuration
Finally, let's look at the PVT model configuration.
- : Patch size at stage
- : Number of output channels at stage
- : Number of encoder layers at stage
- : Reduction ratio of SRA at stage
- : Number of heads in SRA at stage
- : Expansion ratio in the feed-forward layer at stage
The model design follows the design rules of ResNet:
- Use smaller output channels in the shallow stages.
- Concentrate most computational resources in the middle stages.
For discussion purposes, the table below shows a series of PVT models of different scales: PVT-Tiny, PVT-Small, PVT-Medium, and PVT-Large. Their parameter counts are comparable to ResNet18, ResNet50, ResNet101, and ResNet152, respectively.
Discussion
ImageNet Performance
The authors compare PVT with two of the most representative CNN backbones, ResNet and ResNeXt, which are widely used in many downstream task benchmarks.
In the table above, PVT models outperform traditional CNN backbones with similar parameter counts and computational budgets.
For instance, with a similar GFLOP, PVT-Small achieves a top-1 error rate of , outperforming ResNet50's by percentage points ( vs. ).
With similar or lower complexity, PVT models achieve comparable performance to recently proposed Transformer-based models like ViT and DeiT.
PVT-Large has 18.3% top-1 error. Table 2 lists ViT/DeiT-Base/16 at 18.2%, although the paper’s prose says 18.3%. The table supports comparable performance, not exact equality.
Object Detection Performance
Since this architecture targets dense prediction tasks from the outset, let's focus on PVT's performance in dense prediction tasks.
-
Datasets:
- Using the COCO benchmark
- Training set: COCO train2017 (118k images)
- Validation set: COCO val2017 (5k images)
-
Models and Initialization:
- Standard detectors: RetinaNet and Mask R-CNN
- Backbone initialized with ImageNet pretrained weights
- New layers initialized with Xavier initialization
-
Training Settings:
- Batch size: 16
- Hardware: 8 V100 GPUs
- Optimizer: AdamW
- Initial learning rate:
With comparable parameter counts, PVT models significantly outperform traditional models. With RetinaNet, PVT-Tiny's AP is 4.9 points higher than ResNet18 (36.7 vs. 31.8).
With Mask R-CNN, PVT-Tiny's mask AP (APm) is 35.1, 3.9 points higher than ResNet18 (35.1 vs. 31.2), and even 0.7 points higher than ResNet50 (35.1 vs. 34.4).
These results suggest that PVT can serve as a good alternative to CNN backbones for object detection and instance segmentation tasks.
The image below shows PVT's results on the COCO validation set.
Contribution of the Pyramid Structure
The authors conducted several ablation studies to verify the impact of different parts of PVT on performance.
First, the contribution analysis of the pyramid structure is shown in the table below:
With RetinaNet and the COCO 1× schedule, PVT-Small reaches 40.4 AP versus 31.7 for ViT-Small/32, an 8.7-AP difference. ViT-Small/32 draws and interpolates features from several layers to adapt to RetinaNet; ViT-Small/4 ran out of memory on a 32 GB GPU. These are whole-backbone configurations with different patch sizes, SRA and feature scales, so the entire difference cannot be isolated to the pyramid alone.
This indicates that the pyramid structure design helps improve performance in dense prediction tasks.
Depth vs. Width Trade-off
The authors further explore whether PVT should be deeper or wider, as well as the impact of feature map sizes at different stages on performance.
The authors multiplied PVT-Small's hidden dimensions by 1.4 to build a wider version, then compared it with a deeper model of similar parameter count. In this ablation, the deeper model has 44.2M parameters and 41.9 AP, versus 46.8M and 40.8 AP for the wider model. Here, depth performs better under comparable parameter counts.
Addressing but Not Solving the Core Issue
Lastly, the authors discussed performance limitations.
As the input size increases, the GFLOPs of PVT grow faster than ResNet but slower than ViT. When the input size does not exceed 640×640 pixels, PVT-Small and ResNet50 have similar GFLOPs.
Additionally, with the input's shorter side at 800 pixels, RetinaNet with PVT-Small takes 76.9 ms per image on a V100, versus 55.9 ms with ResNet50. Reducing PVT-Small's shorter side to 640 pixels gives 51.7 ms and 38.7 AP, but now the input scales differ.
This identifies a large-image cost for the tested PVT implementation and hardware.
Conclusion
Although in 2021, comparing against ResNet and ResNeXt might seem conservative, this study introduces the pyramid structure into Transformers to provide a pure Transformer backbone for dense prediction tasks, rather than focusing on task-specific heads or image classification models, offering an important direction for future research.
At this point, Transformer-based models in computer vision are still in their early stages, and many potential technologies and applications remain to be explored.