Back to paper library
arXiv 2021ICCV 2021Vision TransformersPaper notes

Swin Transformer

The Dance Begins

Original paper

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Paper authors

Ze Liu · Yutong Lin · Yue Cao …

Show all 8 authors

Ze Liu · Yutong Lin · Yue Cao · Han Hu · Yixuan Wei · Zheng Zhang · Stephen Lin · Baining Guo

Publication
ICCV 2021Conference
First submitted to arXiv
arXiv:2103.14030
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{visiontransformersswintransformer2021,
  title = {{Swin Transformer: Hierarchical Vision Transformer using Shifted Windows}},
  author = {Ze Liu and Yutong Lin and Yue Cao and Han Hu and Yixuan Wei and Zheng Zhang and Stephen Lin and Baining Guo},
  year = {2021},
  eprint = {2103.14030},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.14030}
}
On this page

At the same time, the PVT was proposed, which also incorporated the hierarchical structure of convolutional networks into the Transformer.

PVT uses the SRA mechanism (Spatial Reduction Attention) to reduce the computational cost of the attention mechanism, but this approach doesn't truly solve the problem, as the quadratic complexity of the attention mechanism still exists.

Problem Definition

In this paper, the authors target the original ViT architecture, identifying several key issues:

Multi-scale Detection Capability

Original ViT outputs a single feature scale, commonly from 16×16-pixel patches. It can be adapted for detection and segmentation, but multiscale heads require additional feature-resolution handling and the attention cost can be high.

Quadratic Complexity

For object detection, the input size isn't limited to 224 x 224 images.

Larger input images significantly increase the computational cost of the attention mechanism. For instance, a 224 x 224 image produces 14 x 14 = 196 patches; for a 1536 x 1536 image, the patch count increases to 96 x 96 = 9216. Performing self-attention on this many patches is computationally prohibitive, limiting ViT's application to large images.

Solutions

Model Architecture

To handle the multi-scale problem, a hierarchical architecture is adopted, as shown in the Swin-T architecture above.

An input image of H x W x 3 is split into patches of size 4 x 4, and the feature dimension is projected to C dimensions, resulting in a feature map of size H/4 x W/4 x C. The 48 in the figure represents a patch containing 4 x 4 x 3 pixels, which is then transformed to C dimensions, where C is 96.

Blocks alternate regular window attention (W-MSA) and shifted window attention (SW-MSA) in pairs, so each stage shown has an even number of blocks. Both are Swin blocks; this is not an alternation of global-attention ViT blocks with Swin blocks.

Patch merging concatenates each group of 2×2 neighboring tokens along channels; it does not split one patch into four. The intermediate shape changes from H×W×C to H/2×W/2×4C.

Here’s an illustrative diagram:

Swin V1 applies LayerNorm after concatenation, then a linear projection from 4C to 2C. The projection is equivalent to a 1×1 convolution, but that description must not omit LayerNorm.

Now that we’ve understood the basic architecture, let’s examine the details of the Swin blocks.

Sliding Windows

According to the paper:

  • Gray squares represent tokens. In the first stage each covers 4×4 input pixels; deeper stages merge tokens and cover larger regions.
  • The default window has M×M tokens with M=7. The conceptual figure instead uses M=4 for clarity.

With fixed M=7, each window contains 49 tokens. For N total tokens, all attention matrices contain about 49N elements rather than N². The fuller cost is 4NC2+2M2NC4NC^2+2M^2NC: linear in N for fixed M and C, not linear in image side length.

But this design prevents information flow between windows, hindering the full potential of the Transformer.

Successive layers shift the partition by ⌊M/2⌋\lfloor M/2\rfloor tokens per axis: two for the M=4 illustration, three for the default M=7. New windows cross the previous boundaries and connect neighboring regions.

New Challenges

This method introduces a new challenge:

If the partition boundaries are shifted directly, an N×NN \times N grid may become (N+1)×(N+1)(N+1) \times (N+1) windows of different sizes. The paper’s 2×2 example becomes 3×3; padding each partial window to a fixed size would add attention work.

This is unacceptable for the authors.

To solve this, they propose a special masking method.

Cyclic Shift

Implementation first cyclically shifts the entire feature map toward the top-left, then partitions fixed-size windows. Tokens wrapped from opposite edges must be masked so that originally distant regions do not become artificial neighbors.

This way, we retain the same number of fixed-size windows as regular partitioning instead of padding extra partial windows.

The mask blocks only artificial connections caused by wraparound. Tokens in a genuine shifted window can still interact across old window boundaries; masking all old cross-window connections would defeat the design.

Special Masking

This part is complex, so we refer to the author's explanation on GitHub:

Let’s break down this mask design, as patches in each window need to be aligned for corresponding operations:

  1. The first window is complete, so no attention mask is needed.
  2. Window 1 contains regions 1 and 2 from the Image Mask at left; these regions were not adjacent before the cyclic shift.
  1. Window 2 contains regions 3 and 6, which were not adjacent before the cyclic shift.
  1. Window 3 contains regions 4, 5, 7, and 8.

Reverse Shift

After cyclic shift and special masking, the authors effectively enable information flow between windows while maintaining linear complexity. Finally, a reverse shift restores the original window positions, providing a complete feature map.

Reverse shifting aligns the attention output with the original coordinates for the residual branch and subsequent spatial operations.

Discussion

Performance on ImageNet-1K

For image classification, the authors benchmarked the proposed Swin Transformer on ImageNet-1K.

This dataset contains 1.28 million training images and 50,000 validation images across 1,000 classes. The training setup includes:

  • Optimizer: AdamW
  • Training Epochs: 300
  • Learning Rate Scheduler: Cosine decay
  • Linear Warmup: 20 epochs
  • Batch Size: 1024
  • Initial Learning Rate: 0.001
  • Weight Decay: 0.05
  • Augmentation and Regularization: Includes most strategies except repeated augmentation and EMA, as they did not improve this training setup.

The table above compares different backbone networks, including Transformer and ConvNet-based ones trained on standard ImageNet-1K.

Compared to previous state-of-the-art Transformer architectures (e.g., DeiT), Swin Transformers show significant performance improvements.

  • Swin-T (81.3%) outperforms DeiT-S (79.8%) by 1.5 percentage points, though its listed throughput is lower (755.2 versus 940.4 images/s).
  • Swin-B (83.5%/84.5%) outperforms DeiT-B (81.8%/83.1%) by 1.7 and 1.4 percentage points at 224/384 input, respectively.

Against ConvNet architectures such as RegNet and EfficientNet, accuracy, input size, and throughput vary by configuration. For example, at 224 input Swin-B reaches 83.5% at 278.1 images/s, while RegNetY-16G reaches 82.9% at 334.7 images/s; the table does not show that every Swin configuration is faster. While RegNet and EfficientNet were obtained through architecture search, Swin Transformer is adapted from the standard Transformer and may improve further.

Object Detection Settings

The architecture mainly aims to solve multi-scale issues, making its performance on object detection tasks more significant.

Object detection and instance segmentation experiments are conducted on the COCO 2017 dataset, which includes 118K training images, 5K validation images, and 20K test-dev images.

For ablation studies, the authors consider four typical object detection frameworks:

  1. Cascade Mask R-CNN
  2. ATSS
  3. RepPoints v2
  4. Sparse RCNN

These frameworks share the same training settings:

  • Multi-scale training: Adjust input size so the short side is between 480 to 800 while the long side is up to 1333.
  • Optimizer: AdamW (initial learning rate 0.0001, weight decay 0.05, batch size 16).
  • Training Epochs: 3x schedule (36 epochs).

For system-level comparison, the authors use an improved HTC (HTC++) with:

  • instaboost
  • Stronger multi-scale training
  • 6x schedule (72 epochs)
  • soft-NMS
  • Initialization of the Swin models with ImageNet-22K pretraining

They compare Swin Transformer with standard ConvNets (e.g., ResNe(X)t) and previous Transformer networks (e.g., DeiT), keeping the detection framework and training settings as consistent as possible. Swin Transformer and ResNe(X)t produce hierarchical feature maps suitable for these frameworks, but DeiT produces a single-resolution feature map, so the DeiT comparison also uses a deconvolution layer to construct hierarchical features.

Swin vs ResNe(X)t

Table (a) shows Swin-T and ResNet-50 results on four object detection frameworks.

  • Swin-T provides consistent +3.4∼4.2 box AP improvements over ResNet-50.
  • Model size, FLOPs, and latency are slightly larger.

Table (b) compares Swin Transformer and ResNe(X)t with Cascade Mask RCNN at different model capacities.

  • With Cascade Mask R-CNN, Swin-B achieves 51.9 box AP and 45.0 mask AP, improving by +3.6 box AP and +3.3 mask AP over ResNeXt101-64x4d with similar model size, FLOPs, and latency.

With the improved HTC++ framework on mini-val, ResNeXt101-64x4d scores 52.3 box AP and 46.0 mask AP, while Swin-B scores 56.4 and 49.1, gains of 4.1 and 3.1 AP (see table (c)).

In inference speed, although ResNe(X)t uses highly optimized Cudnn functions while Swin Transformer uses less optimized PyTorch functions, kernel optimization is beyond this paper's scope, so it's not discussed.

Swin vs DeiT

Table (b) shows DeiT-S performance with Cascade Mask R-CNN.

  • Swin-T outperforms DeiT-S by +2.5 box AP and +2.3 mask AP, with similar model size (86M vs. 80M) and significantly higher inference speed (15.3 FPS vs. 10.4 FPS).

DeiT's slower inference speed is mainly due to its quadratic complexity concerning input image size.

Swin vs SOTA

Table (c) compares the best results with previous state-of-the-art models.

  • With ImageNet-22K pretraining and multi-scale testing, the best Swin-L + HTC++ system reaches 58.7 box AP and 51.1 mask AP on COCO test-dev. These are 2.7 and 2.6 AP above the table’s Copy-Paste 56.0 box AP and DetectoRS 48.5 mask AP, respectively. This is a system-level comparison across methods, unlike the backbone ablation. “No external data” qualifies the compared Copy-Paste setting, not an absence of external pretraining for Swin.

Through these comparisons and experimental results, the authors conclude:

  • Swin Transformer significantly outperforms ResNe(X)t and DeiT in multiple object detection frameworks.
  • In the roughly matched model-size comparisons of table (b), Swin Transformer improves detection and instance segmentation accuracy; costs across different frameworks and system settings need separate comparisons.
  • Despite current inference speed limitations due to less optimized PyTorch functions, Swin Transformer demonstrates strong performance potential.

Ablation Studies

Finally, the authors detail ablation studies on Swin Transformer for image classification and object detection tasks to verify the effectiveness of key design elements.

  1. Shifted Windows

    The authors first investigate the effect of shifted windows.

    Experiments show that Swin-T with shifted windows outperforms fixed-window partitioning across all tasks:

    • On ImageNet-1K, top-1 accuracy improves by 1.1 percentage points.
    • On COCO, box AP improves by 2.8, and mask AP by 2.2.
    • On ADE20K, mIoU improves by 2.8.

    These results highlight shifted windows' ability to effectively establish connections between windows in early layers. Moreover, the latency overhead from shifted windows is minimal.

  2. Relative Position Bias

    Next, the authors compare different positional embedding methods. Results show that Swin-T with relative position bias significantly outperforms methods without positional encoding and absolute positional embedding:

    • On ImageNet-1K, top-1 accuracy improves by 1.2 and 0.8 percentage points.
    • On COCO, box AP improves by 1.3 and 1.5, mask AP by 1.1 and 1.3.
    • On ADE20K, mIoU improves by 2.3 and 2.9.

    This indicates that relative position bias is effective across image classification, object detection, and semantic segmentation tasks.

  3. Different Self-Attention Methods

    Finally, the authors compare practical speeds of different self-attention computation methods.

    Compared with padding partial shifted windows directly, cyclic shift raises whole-model throughput from 670/371/236 to 755/437/278 FPS for Swin-T/S/B, about 13%/18%/18% faster. The table also tests both naive and optimized-kernel sliding windows; the speed comparison depends on the implementation.

    The authors also compare shifted windows with Performer using a Swin-T backbone: whole-model throughput is 755 versus 638 FPS in table 5, and ImageNet-1K top-1 accuracy is 81.3% versus 79.0% in table 6, a 2.3-point difference. Sliding windows reach a similar 81.4% top-1 accuracy, but both implementations in table 5 are slower.

Conclusion

Swin Transformer is an innovative vision transformer that generates hierarchical feature representations with linear computational complexity concerning input image size. It demonstrates state-of-the-art performance on COCO object detection and ADE20K semantic segmentation tasks, significantly outperforming previous best methods.

Finally, the authors highlight the key role of shifted windows in Swin Transformer and anticipate its application in NLP, potentially becoming a general-purpose transformer model.