Back to paper library
arXiv 2023ICCV 2023ReparameterizationPaper notes

FastViT

Reparameterized ViT Experiments

Original paper

FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization

Paper authors

Pavan Kumar Anasosalu Vasu · James Gabriel · Jeff Zhu · Oncel Tuzel · Anurag Ranjan

Publication
ICCV 2023Conference
First submitted to arXiv
arXiv:2303.14189
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{reparameterizationfastvit2023,
  title = {{FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization}},
  author = {Pavan Kumar Anasosalu Vasu and James Gabriel and Jeff Zhu and Oncel Tuzel and Anurag Ranjan},
  year = {2023},
  eprint = {2303.14189},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.14189}
}
On this page

After the recent introduction of MobileOne, Apple continues to explore more possibilities with model reparameterization. Previously, they tackled the MobileNet architecture; now, it's Vision Transformer's turn.

Problem Definition

Several design directions have been established over the past few years for Transformer models in computer vision.

First, we structure multiple layers corresponding to the multi-scale resolution needs of vision tasks. This concept comes from PVT and Swin Transformer:

Next, we design a Token Mixer block paired with an FFN block to form a basic module, a concept derived from MetaFormer:

After proposing MobileOne, the authors turned their attention to this structural architecture.

Is there an opportunity to reparameterize the Vision Transformer series of models as well?

Solution

Model Architecture

The FastViT architecture comprises four distinct stages, operating at different scales.

  • RepMixer

    RepMixer is based on the convolutional mixing concept introduced by ConvMixer, with its mixing block defined as follows:

    Y=BN(σ(DWConv(X)))+XY = \text{BN}(\sigma(\text{DWConv}(X))) + X

    Where σ\sigma represents a non-linear activation function, BN denotes Batch Normalization, and DWConv is a depth-wise convolution layer. FastViT redesigns the block by rearranging operations and removing this nonlinearity, giving the conceptual form:

    Y=DWConv(BN(X))+XY = \text{DWConv}(\text{BN}(X)) + X

    This architectural change requires training and is not equivalent to the original nonlinear ConvMixer. Once trained, the mergeable operations can be fused for inference into a depthwise convolution; “fused” denotes changed weights and bias:

    Y=DWConvfused(X)Y = \text{DWConv}_{\mathrm{fused}}(X)

    This reparameterization removes residual-addition and BN overhead and reduces memory access. Compared to PoolFormer’s PoolMixer, RepMixer offers better inference performance.

  • Positional Encoding

    FastViT employs conditional positional encoding (referenced below), which dynamically generates encodings based on the local neighborhood of input tokens. These encodings are generated through depth-wise convolutions and added to the patch embedding. Due to the lack of non-linear operations in this process, this block can be effectively reparameterized.

  • Depth-wise Separable Convolution

    Continuing the design philosophy of MobileOne, FastViT replaces dense k×kk \times k convolutions with depth-wise separable convolutions, i.e., k×kk \times k depth-wise convolutions and 1×11 \times 1 point-wise convolutions, paired with reparameterized training strategies.

    This approach enhances model capacity without significantly increasing training time. For instance, FastViT-SA12 and FastViT-SA36 saw training time increases of only 6.7% and 4.4%, respectively.

  • Large Kernel Convolution

    To address the local receptive field limitation of RepMixer compared to self-attention Token Mixers, FastViT integrates depth-wise large kernel convolutions within FFN and patch embedding layers.

    In the large-kernel ablation, top-1 accuracy rises from 78.9% to 79.8% for V1→V4 and from 79.9% to 80.6% for V2→V5. Mobile latency rises by 0.1 ms in each comparison. This table compares accuracy and latency; it does not separately test robustness.

Model Configurations

FastViT offers various configurations to meet different computational demands and application scenarios, as shown in the table above. These configurations vary in embedding dimensions, MLP expansion ratios, and the types of modules included.

Detailed specifics are as follows:

  • Models with smaller embedding dimensions, such as [64, 128, 256, 512], are prefixed with “S”.
  • Models including self-attention (Self-Attention) layers are prefixed with “SA”.
  • Models with larger embedding dimensions, such as [76, 152, 304, 608], are prefixed with “M”.
  • Models with MLP expansion ratios less than 4 are prefixed with “T”.

The numbers in the table represent the total number of FastViT modules, with FLOP counts calculated using the fvcore library.

Training and Evaluation Strategy

The authors conducted experiments using the ImageNet-1K dataset, which includes approximately 1.3 million training images and 50,000 validation images.

Base training uses 256×256 inputs, with the following settings:

  • Optimizer: AdamW
  • Training epochs: 300
  • Weight decay: 0.05
  • Peak learning rate: 10−310^{-3}
  • Total batch size: 1024
  • Warm-up epochs: 5
  • Learning rate decay: Cosine annealing strategy
  • Implementation tool: timm library
  • Training hardware: 8 NVIDIA A100 GPUs

FastViT-SA36 and MA36 were fine-tuned for 30 epochs at 384×384, with these settings:

  • Weight decay: 10−810^{-8}
  • Learning rate: 5×10−55 \times 10^{-5}
  • Batch size: 512

The authors compared FastViT with other state-of-the-art models on the ImageNet-1K dataset. For fair comparison, they modified the official implementation of ConvNeXt to avoid expensive reshape operations. Additionally, due to poor support for deformable convolutions in the two libraries, LITv2 could not be reliably exported in this paper.

Discussion

Performance on ImageNet

The comparison uses Core ML Tools 6.0, iOS 16 and batch size 1 on mobile. The experimental setup specifies iPhone12 Pro Max, while captions use iPhone12 Pro. GPU measurements use RTX 2080Ti, TensorRT 8.0.1.6 and batch size 8, so they are not single-image GPU latency. Models use the input sizes listed in the table:

  • FastViT-MA36:

    • Achieved a Top-1 accuracy of 84.9% at 384×384
    • 384×384 inputs; 42.7M parameters versus LITv2-B’s 87.0M, about 50.9% fewer when recalculated from the table
    • Consumes 55.4% fewer FLOPs
  • FastViT-S12:

    • About 26.3% lower mobile latency than MobileOne-S4
    • 2.2 ms versus 2.6 ms GPU latency for MobileOne-S4, about 15% lower using the table values
  • FastViT-MA36:

    • Top-1 accuracy of 83.9% at 256×256
    • 1.9 times faster than the optimized ConvNeXt-B model on iPhone 12 Pro
    • 2.0 times faster on GPU
  • FastViT-MA36:

    • Top-1 accuracy of 84.9% at 384×384
    • 16.5 ms versus 16.9 ms for NFNet-F1 on GPU, close in this benchmark
    • 42.7M parameters versus NFNet-F1’s 132.6M, about 67.8% fewer when recalculated from the table
    • Uses 50.1% fewer FLOPs
    • Mobile latency reduced from 16.1 ms to 9.2 ms, about a 42.8% reduction

Conclusion

FastViT achieves a balance of high performance and low latency through structural reparameterization and other improvements, demonstrating robust performance across various application scenarios.