MetaFormer Baselines for Vision
- Publication
- IEEE Transactions on Pattern Analysis and Machine Intelligence 2024Journal
- First submitted to arXiv
- arXiv:2210.13452
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{visiontransformerscaformer2022,
title = {{MetaFormer Baselines for Vision}},
author = {Weihao Yu and Chenyang Si and Pan Zhou and Mi Luo and Yichen Zhou and Jiashi Feng and Shuicheng Yan and Xinchao Wang},
year = {2022},
eprint = {2210.13452},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2210.13452}
}
After publishing the first MetaFormer paper, the authors released another paper to demonstrate its performance.
This paper builds on that framework and shows us how to assemble models with it. After reading it, we think of it as the MetaFormer user manual. Paper Figure 1 puts the answer up front: identity versus random mixing on the left, convolution versus convolution-plus-attention on the right. Bubble area indicates parameter count, the horizontal axis is MACs, and the vertical axis is ImageNet-1K top-1 accuracy. MACs are not measured runtime.
Defining the Problem
The authors guide us step by step through using MetaFormer. More precisely, PoolFormer showed that simple pooling works within this skeleton. What happens if the mixer does nothing, uses a frozen random matrix, or switches to conventional convolution or attention? How much of the outcome comes from the mixer, and how much from normalization, residuals, activations, and the training recipe? The paper builds baselines to examine these questions. It does not claim that a newly invented mixer solves them all.
Solving the Problem
Model Architecture
Paper Figure 2 shows four-stage arrangements on the left and the mixer inside each block on the right. The authors try four token mixers: identity mapping, a frozen random matrix, MobileNetV2-style inverted separable convolution, and the original Transformer's self-attention. The drawing is more useful than memorizing names: check the stages first, then note that the residuals and channel MLP remain.
- IdentityFormer: all four stages use identity mapping. There is no cross-token mixing in these blocks; the other parts of MetaFormer remain.
- RandFormer: the first two stages use identity mapping; the last two use a globally random matrix frozen after initialization, , computing . It is neither input-dependent nor a query–key attention map. Parameters and computation grow roughly quadratically with token count, so the authors put it only in the later, shorter-token stages.
- ConvFormer: every stage uses a MobileNetV2-style separable convolution: pointwise, depthwise, pointwise. Its depthwise kernel is 7 and expansion ratio is 2. This is a convolutional model without self-attention, and the mixer is more than a single depthwise layer.
- CAFormer: the first two stages use that convolution; the last two use vanilla self-attention. Local, cheaper processing handles many early tokens, while global mixing happens after downsampling.
StarReLU
Beyond the basic architecture, the authors introduce StarReLU. Let us start with the familiar ReLU and GELU:
is the standard Gaussian cumulative distribution function. Under the paper's FLOP counting, ReLU costs 1 FLOP per element and the GELU approximation 14. These are counts for the activation itself, not the whole network. To simplify GELU, earlier work tried Squared ReLU:
Squared ReLU is straightforward:
It costs 2 FLOPs per element in the paper's count. Yet on some image-classification models it does not match GELU, possibly because its output distribution shifts. If , then
The authors first derive a standardized form:
Actual network inputs need not follow a standard normal distribution. Thus the scale and offset become learnable scalars shared across channels:
The paper estimates 4 FLOPs per element, about 71% fewer activation FLOPs than GELU's 14. The name points to its repeated multiplications. Actual speed is another matter: the paper's PyTorch API timing found unfused StarReLU even slower than GELU. Kernel implementation and device matter; 71% is not a whole-network speedup or a latency measurement.
Discussion
Using Basic Token Mixers
Paper Table 3 compares identity, random mixing, and PoolFormerV2 on ImageNet-1K at . PoolFormerV2 keeps the PoolFormer architecture but applies this paper's updated design/training techniques for comparison. Its numbers should not be silently substituted into the earlier PoolFormer experiment.
IdentityFormer-S12 reaches 74.6%, above RSB-ResNet-18 at 70.6%; S24 reaches 78.2%, above RSB-ResNet-34 at 75.5%. Scaling IdentityFormer-M48 to 73.3M parameters and 11.5G MACs gives 80.4%, above RSB-ResNet-50's 79.8%, though below the table's RSB-ResNet-101 at 81.3% and ResNet-152 at 81.8%. The useful claim is that a model with no explicit token mixing still learns substantial representations, not that any model with similar resources is guaranteed at least 80%.
RandFormer-S12/M48 report 76.6/81.4%, up 2.0/1.0 percentage points over their IdentityFormer counterparts. RandFormer-M36 also reaches 81.2%, close to RSB-ResNet-101's 81.3%. S12, however, adds 0.2M frozen matrix parameters and M48 0.9M, with higher MACs. PoolFormerV2-S12/M48 remain higher at 78.0/82.6%. Frozen random mixing helps, while local pooling's inductive bias is useful in these settings too.
Using Common Token Mixers
Paper Table 5 groups models by size and separately reports performance at and, for some, after fine-tuning to . Compare like columns. ConvFormer-B36 obtains 84.8% with 100M parameters and 22.6G MACs; ConvNeXt-L gives 84.3% with 198M and 34.4G. Thus the paper's 0.5-point advantage with 51% of the parameters and 66% of the MACs is over ConvNeXt-L, not ConvNeXt-B. ConvNeXt-B is a different row at 83.8%/89M/15.4G. ConvFormer-M36 also leads Swin-B/CoAtNet-2 by 1.0/0.4 percentage points, using 35%/24% fewer parameters and 17%/18% fewer MACs in the listed comparison.
At , CAFormer-B36 reaches 85.5% with 99M parameters and 23.2G MACs, versus 85.3%/218M/42.1G for MViTv2-L in the same table. This was a record at the paper's time and under ImageNet-1K supervised training without external data or distillation, not a current claim across every training regime. After ImageNet-21K pretraining, paper Table 7 reports 87.0% for ConvFormer-B36 and 87.4% for CAFormer-B36 at , 2.2/1.9 points above their ImageNet-1K-only results. Under that same Table 7 setting, ConvFormer-B36 exceeds ConvNeXt-L's 86.8% by 0.2 points, with 49% fewer parameters and 34% fewer MACs. These are separate data conditions from Table 5. The column also entails extra high-resolution fine-tuning and is not a same-cost comparison with .
CAFormer balances local convolutions while tokens are numerous against attention after downsampling. Table 5 includes different models and training recipes, so it shows this engineering combination is competitive; it does not isolate a single mixer as the sole cause. For large-image detection, the paper also restricts CAFormer's attention to sliding windows to avoid running out of memory. “Global attention in later stages” therefore has a condition when transferred to that task.
Ablation Experiments
The baseline in paper Table 8 uses StarReLU: ConvFormer-S18 reaches 83.0%, CAFormer-S18 83.6%. Its arrows mean “replace StarReLU with ReLU/Squared ReLU/GELU,” not the other direction. ReLU yields 82.1/82.9%, Squared ReLU 82.6/83.4%, and GELU 82.7/83.4%. Thus StarReLU is 0.3/0.2 points ahead of GELU for these two S18 models. The default uses learnable scale and bias; the table also varies fixed or learned forms.
ResScale performs better here than omitting scaling or switching to LayerScale. Enabling biases within the blocks leaves ConvFormer-S18 almost unchanged and moves CAFormer-S18 from 83.6% to 83.5%. These small choices, together with StarReLU, form a better training baseline. They should not be credited entirely to the mixer.
Conclusion
The authors use this paper to show what MetaFormer can do. I take four points from their “user manual”:
- MetaFormer offers a strong baseline. Even with identity mapping as its mixer, IdentityFormer-M48 reaches 80.4% on ImageNet-1K. That is a result for this particular model and training setting.
- It pairs well with different mixers. A frozen random mixer lifts RandFormer-M48 to 81.4%, above its IdentityFormer counterpart. That makes the framework a useful starting point when trying a new mixer, although Table 3 still favors PoolFormerV2 at the same scale.
- Familiar mixers can produce strong results. ConvFormer uses separable convolutions and outperforms the listed ConvNeXt comparison; CAFormer combines them in the early stages with later self-attention and reached 85.5% at under the paper's ImageNet-1K supervised setting, without extra data or distillation. The record is a paper-time claim.
- StarReLU is worth trying. As a Squared ReLU variant intended to reduce distribution shift, it cuts the paper's activation FLOP count by 71% relative to GELU and improves the two S18 results in Table 8. That FLOP count is not a measured end-to-end latency gain.
MetaFormer shows considerable room for stronger models. With better mixers or training strategies, we can expect more good results from this framework; the user manual is a place to start building and checking them.