Back to paper library
arXiv 2021ICLR 2023Vision TransformersPaper notes

CPVT

Conditional Positional Encodings

Original paper

Conditional Positional Encodings for Vision Transformers

Paper authors

Xiangxiang Chu · Zhi Tian · Bo Zhang · Xinlong Wang · Chunhua Shen

Publication
ICLR 2023Conference
First submitted to arXiv
arXiv:2102.10882
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{visiontransformerscpvt2021,
  title = {{Conditional Positional Encodings for Vision Transformers}},
  author = {Xiangxiang Chu and Zhi Tian and Bo Zhang and Xinlong Wang and Chunhua Shen},
  year = {2021},
  eprint = {2102.10882},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.10882}
}
On this page

Shortly after the introduction of ViT (Vision Transformer), a wave of research emerged to improve its architecture.

If you've worked with ViT-based models, you might share the same frustration: the "learnable" positional encodings are challenging to use!

Defining the Problem

In ViT, the input image is divided into fixed-size patches, and each patch is combined with a corresponding learnable absolute positional encoding. However, this positional encoding approach comes with two significant issues:

Sequence Expansion

The length of the absolute positional encoding is fixed, which limits the model’s ability to handle longer sequences than what it was trained on.

You might wonder: Can we just discard the positional encoding altogether?

  • The answer is no!

In the CPVT paper’s DeiT-tiny ablation, removing position encodings lowers accuracy from 72.2% to 68.2%, a 4-point loss. This argues against simply removing them in that setup, not against every alternative architecture.

This makes sense because the order of the input sequence carries crucial information. Without positional encodings, the model would lose the ability to capture this order.

Translation Equivariance

Translation equivariance means the output feature map moves with the input. Classification instead usually aims for translation invariance: retaining the same class prediction.

Fixed absolute positions disrupt feature equivariance because a moved patch receives a different embedding. PEG supplies a convolutional bias toward equivariance, but zero-padding boundaries and patch extraction prevent a strict guarantee for arbitrary pixel shifts.

Research on DeiT has demonstrated that positional encodings can be extended to longer sequences using interpolation. However, this method requires multiple rounds of fine-tuning. If the model is deployed without fine-tuning, performance drastically declines.

What about using relative positional encodings (RPE) instead?

  • The answer: It addresses both of these issues.

The RPE configuration tested here lacks absolute position cues and performs below APE.

Solving the Problem

Conditional Positional Encodings

To address these issues, the authors propose a new type of positional encoding: Conditional Positional Encodings (CPE).

As shown in the figure above, the leftmost image represents the original ViT architecture, where the image patches are directly added to a "learnable" absolute positional encoding.

In the center image, the proposed CPE architecture introduces a key modification: the positional encoding calculation occurs within the Transformer encoder itself.

Specifically, it’s applied after the first encoder layer but before the second one.

Positional Encoding Generator

The Positional Encoding Generator (PEG) is key to the proposed CPE. Let’s break down its structure:

  • It’s simply a 2D convolution layer!

After patch embedding and the first encoder block, the feature tokens (excluding the class token) are arranged back into a 2D grid. A convolution then generates positional encodings from that grid, which are added to the feature tokens.

This convolution layer uses zero padding. Border tokens therefore encounter padded zeros, providing some absolute-position cues; this also means the operation is not strictly translation-equivariant. Through convolution, the model dynamically generates positional encodings from each token's local neighbors—hence Conditional Positional Encoding.

Below is a standalone, executable PEG component adapted from the paper’s sketch. It preserves CLS while adding depthwise-convolution output to a 2D patch grid; it is not a complete CPVT classifier.

python
import torch
import torch.nn as nn

class PEG(nn.Module):
    def __init__(self, dim=192, k=3):
        super().__init__()
        if k < 3 or k % 2 == 0:
            raise ValueError("k must be odd and at least 3")
        self.pos = nn.Conv2d(dim, dim, k, padding=k // 2, groups=dim)

    def forward(self, x, height, width):
        # x: [batch, 1 + height * width, channels]; first token is CLS.
        batch, tokens, channels = x.shape
        if tokens != 1 + height * width:
            raise ValueError("Patch grid and token count do not match")
        cls_token, patches = x[:, :1], x[:, 1:]
        grid = patches.transpose(1, 2).reshape(batch, channels, height, width)
        patches = (grid + self.pos(grid)).flatten(2).transpose(1, 2)
        return torch.cat((cls_token, patches), dim=1)

"""Apply after the first encoder block; this is the PEG component, not a full ViT."""
x = torch.randn(2, 1 + 14 * 14, 192)
y = PEG(dim=192)(x, 14, 14)
assert y.shape == x.shape
assert torch.equal(y[:, :1], x[:, :1])

PEG consists of a depthwise Conv2d plus a residual addition. The class token does not participate in the spatial convolution.

Model Training Strategy

To validate this new positional encoding approach, the authors report using the DeiT training strategy. The main text's experimental setup is as follows:

  1. Dataset

    • The study uses the ImageNet dataset, which includes 1,000 classes and 1.3 million images for training.
    • Evaluation is conducted on the 50,000-image validation set.
    • The larger JFT-300M dataset was not used.
  2. Model Variants

    • Three model variants of different sizes were designed for various computational needs.
    • All experiments were run on Tesla V100 machines.
    • Training the tiny model for 300 epochs took approximately 1.3 days (on a single node with 8 V100 GPUs), while CPVT-S and CPVT-B took 1.6 days and 2.5 days, respectively.
  3. Training Details

    • Section 4.1 reports that, except for CPVT-B, the models were trained for 300 epochs on Tesla V100 machines with a global batch size of 2048.
    • That section reports using the AdamW optimizer.

Discussion

PEG Placement

Why is PEG placed after the first encoder layer and before the second?

The authors tried PEG at different positions. In the left table, position 0 reaches 72.4% top-1 accuracy and position 3 reaches 72.3%, the two best results. Position -1 reaches 70.6%, below position 0.

The difference is the features available to PEG: at position 0, they have passed through one global self-attention block; at position -1, a small convolution can see only local regions. The authors reasoned that a larger kernel at position -1 might narrow the gap.

To verify this hypothesis, the authors used a larger convolution kernel (27×27, with padding size 13) at position -1. The results showed similar performance (72.5%) to position 0, supporting this hypothesis.

However, increasing the convolution kernel size also increases computational complexity. Therefore, the authors chose to place PEG at position 0 to achieve better performance without added complexity.

Where Does PEG’s Improvement Come From?

One might suspect that PEG's improvement comes merely from extra learnable convolution parameters, rather than local relationships. First, the authors applied PEG only when calculating Q and K in the attention layer, so its signal reached attention as positional information.

Under that Q/K-only condition, DeiT-tiny reaches 71.3% top-1 accuracy, above the 68.2% no-position baseline and close to the usual 72.4% PEG result. The authors take this as evidence that PEG works as positional encoding, rather than only by adding parameters.

Separately, a randomly initialized 3×3 PEG with its weights frozen during training also reaches 71.3%, 3.1 percentage points above the 68.2% no-position baseline. A PEG initialized from learned weights and then frozen reaches 72.3%, close to the trainable PEG's 72.4%. Since the fixed weights are not updated, these results support the role of local spatial cues without isolating every possible contributing mechanism.

Conversely, when using 12 convolution layers (with kernel size 1, i.e., no local relationships) to replace PEG, the number of learnable parameters far exceeded that of PEG, but performance only improved by 0.4 percentage points, reaching 68.6%. This further confirms that PEG’s effectiveness comes mainly from positional information, not the extra parameters.

Generalization to Higher Resolutions

As shown in the table above, PEG can directly generalize to larger image sizes without requiring fine-tuning. This was tested by evaluating models trained on 224 × 224 images with larger resolutions (384 × 384, 448 × 448, 512 × 512).

For DeiT-tiny at 384 × 384, accuracy falls from 72.2% to 71.2% with learnable position encodings. With sinusoidal encodings, it falls from 72.3% to 70.8%.

With one PEG, CPVT-Ti improves from 72.4% at 224 to 73.2% at 384. With five PEGs, it improves from 73.4% to 74.2%, 3.0 points above DeiT-tiny at 384. The five-PEG model then falls to 72.6% at 448 and 70.8% at 512, while remaining above DeiT-tiny at those resolutions.

Reducing Complexity

In PEG, if depth-wise convolutions are used, the additional number of parameters is d×l×k2d \times l \times k^2 (where dd is the model dimension, ll is the depth of the convolution, and kk is the kernel size). For separable convolutions, the number of parameters is l(d2+k2d)l(d^2 + k^2d).

For example, in CPVT-Tiny, when k=3k = 3 and l=1l = 1 (i.e., the model dimension d=192d = 192), PEG introduces 1,728 parameters. In contrast, the DeiT-tiny model uses learnable positional encodings with 192 × 14 × 14 = 37,632 parameters.

The single depthwise PEG has 35,904 fewer weights than the patch-position table. These counts exclude biases and any class-token position entry; the example Conv2d includes bias and has 1,920 parameters. The formula gives 38,592 weights for one separable convolution, 960 more than 37,632. Four layers require 154,368 weights, so the single-layer difference cannot be reused. These positional-encoding weights are still small beside a roughly 5.7M-parameter model.

Performance on ImageNet

In the reported setup, most CPVT configurations have higher ImageNet top-1 accuracy than the DeiT entries at similar V100 throughput. Table 2 showed improved accuracy at 384×384 without fine-tuning; further enlarging the input to 448 or 512 lowers CPVT-Ti's own accuracy, though it remains above DeiT-tiny at the same resolution.

Replacing CLS with global average pooling raises the five-PEG CPVT-Ti from 73.4% to 74.9%, versus 72.2% for DeiT-tiny without distillation. This is a result within the paper’s historical setting, not a current record for all ViTs.

In a separate comparison, both CPVT-Tiny and DeiT-tiny received additional hard-distillation training with RegNetY-160 as teacher. CPVT reaches 75.9% top-1 accuracy, 1.4 percentage points above the distilled DeiT-tiny's 74.5%.

Comparison with Other Positional Encodings

Finally, the authors compared PEG with other common positional encoding methods and analyzed the combination of PEG with absolute positional encoding (APE) and relative positional encoding (RPE).

  1. Comparison of PEG and Other Positional Encodings

    • The DeiT-tiny model using learnable absolute positional encoding (LE) achieved a Top-1 accuracy of 72.2%.
    • Using 2D sinusoidal encoding resulted in similar performance.
    • When using relative positional encoding (RPE, with hyperparameter K set to 8), the Top-1 accuracy dropped to 70.5%, lower than LE. One reason for this is that RPE does not encode absolute position information, leading to reduced performance.
  2. Combining PEG with Other Encodings

    Combining learnable absolute positional encoding with a single PEG raises CPVT-Ti from 72.4% to 72.9%. Stacking four PEGs at that same position while retaining the learnable encoding also gives 72.9%. Placing one PEG after each of the first five Transformer blocks, without that learnable encoding, reaches 73.4%.

Conclusion

PEG generates position information from input features and supports different patch-sequence lengths. The results support its local spatial bias and generalization; claims of equivariance still depend on padding and boundary conditions.

Furthermore, PEG is simple to implement, introduces minimal additional computational costs, and requires only a small number of extra parameters, making it an effective and practical solution.