MetaFormer Is Actually What You Need for Vision
- Publication
- CVPR 2022Conference
- First submitted to arXiv
- arXiv:2111.11418
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{visiontransformerspoolformer2021,
title = {{MetaFormer Is Actually What You Need for Vision}},
author = {Weihao Yu and Mi Luo and Pan Zhou and Chenyang Si and Yichen Zhou and Xinchao Wang and Jiashi Feng and Shuicheng Yan},
year = {2021},
eprint = {2111.11418},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2111.11418}
}
The introduction of the Transformer architecture into the vision domain has sparked a significant wave of research. In previous studies, most researchers focused on improving the performance of the Token-Mixer or optimizing the Patchify process. However, an important issue was often overlooked: the framework.
Problem Definition
Start with Figure 1 of the paper. The leftmost block leaves the token mixer unspecified. Fill that slot with attention, a spatial MLP, or pooling, and you get three different model families. The two residual paths, normalization, and channel MLP remain. The authors call that shared skeleton MetaFormer.
Input Embedding
This turns an image into a sequence of tokens, often using a convolution. PoolFormer uses these embeddings in a hierarchical design and downsamples later feature maps.
Token-Mixer
This component mixes information across tokens, usually through Transformer self-attention. Recent MLP-Mixer work showed that a spatial MLP can also fill the empty box in Figure 1, although its receptive field and cost differ; replacing attention does not automatically save computation at every input size.
Channel MLP
A two-layer MLP mixes the channels within each token. It works alongside cross-position mixing. The two plus signs in Figure 1 matter too: residual paths let signals bypass the mixer and the channel MLP.
Is Token-Mixer Really the Key?
The authors ask whether attention-based token mixing contributes most to the Transformer's success, or whether the overall skeleton—the MetaFormer—matters more. To test it, they replace attention with simple local average pooling and see how far the complete model gets. The experiment tests whether a complex mixer is necessary; it does not isolate every contributor to accuracy.
Solution
Model Architecture
To test MetaFormer, the authors deliberately weaken the token mixer by replacing attention with pooling—about as simple an operation as one can use short of the identity mapping. Each position gathers features from nearby tokens. Paper Figure 2(a) shows four hierarchical stages: the first reduces the spatial size to ; the following stages reduce it to , , and . In Figure 2(b), each block applies normalization and a pooling mixer, then normalization and a channel MLP, with a residual connection around each sub-block.
Following the hierarchical idea used in PVT, the early stages process more tokens and later stages work on smaller feature maps. The deliberately simple mixer uses average pooling at stride 1, not max pooling. Algorithm 1 gives the essential code:
import torch.nn as nn
class Pooling(nn.Module):
def __init__(self, pool_size=3):
super().__init__()
self.pool = nn.AvgPool2d(
pool_size, stride=1, padding=pool_size // 2,
count_include_pad=False,
)
def forward(self, x):
return self.pool(x) - x
The self.pool(x) - x line may look odd at first. Let be the tensor entering the mixer. The block's first residual path actually computes
The subtracted tensor is ; the residual adds back . Simplifying this to would lose the distinction across normalization. The subtraction removes the mixer's self term; the channel MLP and its residual still operate afterward. Pooling itself has no trainable weights, but the full PoolFormer certainly has trainable embeddings and MLP weights.
Model Configuration
Paper Table 1 lists five sizes from S12 to M48. Stage depths follow : for example, S12 has 2/2/6/2 blocks. Stage 1 uses a , stride-4 patch embedding; later stages use , stride-2 downsampling. The channel MLP expansion ratio is 4. The S models use widths 64/128/320/512; the M models use 96/192/384/768. Read depth, width, and MACs together: pooling is parameter-free, while the other modules retain learned parameters.
Discussion
ImageNet Experiments
The PoolFormer results are striking. In the imported ImageNet-1K table, S24 reaches 80.3% with 21M parameters and 3.4G MACs, above DeiT-S at 79.8% and ResMLP-S24 at 79.4%. S36 reaches 81.4% with 31M parameters and 5.0G MACs, slightly above PVT-Medium at 81.2%. These are comparisons within the table; training recipes across papers are not identical.
Plotting the same results makes the pattern easier to see: in these comparisons PoolFormer often gets higher accuracy with fewer parameters and MACs, though other models occupy different cost and accuracy positions. The left horizontal axis is MACs, the right one is parameter count; neither is measured latency.
Ablation Experiments
Paper Table 5 uses PoolFormer-S12 at 77.2% as a baseline and changes one design group at a time. It addresses the “can the skeleton hold up?” question more directly than comparisons across different papers.
- Change the mixer. Identity mapping still reaches 74.3%, 2.9 percentage points below baseline. A frozen global random matrix reaches 75.8%, 1.5 points above identity, but adds about 21M frozen parameters not counted in the table's 11.9M and more MACs. Depthwise convolution reaches 78.1%, 0.9 point above PoolFormer-S12. A simple mixer can work; the choice still matters.
- Change the pooling kernel. Sizes 3, 5, and 7 give 77.2/77.2/77.1%; size 9 falls to 76.8%. Size 3 is a reasonable default here.
- Change normalization. Modified Layer Normalization (MLN) computes moments across token and channel dimensions. Replacing it with ordinary LayerNorm or BatchNorm gives 76.5/76.4%; removing normalization gives only 46.1%. This is a result for this model and setting, not a universal ranking of normalization layers.
- Change activation. ReLU drops the GELU baseline from 77.2% to 76.4%; SiLU remains at 77.2%.
- Remove residuals or the channel MLP. Accuracy falls to 0.1/5.7%, respectively. Both are important under this training setup. The 74.3% identity-mixer configuration still uses residuals and the channel MLP.
- Mix stages. The authors try pooling in the lower stages and attention or a spatial MLP in the upper stages. Pooling in the first two stages and attention in the last two yields 81.0% with 16.5M parameters and 2.5G MACs in the imported ablation table. ResMLP-B24 also has 81.0% in the main table but uses 116M parameters and 23G MACs. This is an accuracy/resource comparison, not an experiment controlling all differences between studies.
From these ablations, the authors argue that MetaFormer itself contributes strongly, simple pooling can be competitive, their modified LayerNorm beats ordinary LayerNorm and BatchNorm in this setting, and residuals and the channel MLP are indispensable here. Combining pooling with other token mixers may improve results further. The framework matters; the mixer still has work to do.
Conclusion
In this study, the authors abstract the Transformer into a general architecture: MetaFormer. Attention becomes one possible token mixer, rather than a required part of the framework.
By using something as simple as pooling for token mixing, they show that PoolFormer can compete on visual tasks, lending weight to the bold claim that MetaFormer Is Actually What You Need for Vision.
One direction for future work is to test how broadly and reliably MetaFormer applies. The authors also suggest trying PoolFormer on natural language processing tasks. If that works, the case for MetaFormer as a general architecture will be stronger.