{"notes":[{"id":"vision-transformers/caformer","category":"vision-transformers","legacyPath":"vision-transformers/2210-caformer","legacySlug":"2210-caformer","legacyPaths":[],"title":"CAFormer","subtitle":"MetaFormer User Manual","author":"Z. Yuan","paperDate":"2022-10-24","publishedOn":"2024-07-11","paperTitle":"MetaFormer Baselines for Vision","paperUrl":"https://doi.org/10.1109/TPAMI.2023.3329173","bibliography":{"title":"MetaFormer Baselines for Vision","authors":["Weihao Yu","Chenyang Si","Pan Zhou","Mi Luo","Yichen Zhou","Jiashi Feng","Shuicheng Yan","Xinchao Wang"],"url":"https://doi.org/10.1109/TPAMI.2023.3329173","pdfUrl":"https://arxiv.org/pdf/2210.13452v4","codeUrl":"https://github.com/sail-sg/metaformer","preprint":{"id":"2210.13452","submittedOn":"2022-10-24","url":"https://arxiv.org/abs/2210.13452"},"publication":{"venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","shortVenue":"IEEE TPAMI","year":2024,"kind":"journal","url":"https://doi.org/10.1109/TPAMI.2023.3329173"},"citation":"@misc{visiontransformerscaformer2022,\n  title = {{MetaFormer Baselines for Vision}},\n  author = {Weihao Yu and Chenyang Si and Pan Zhou and Mi Luo and Yichen Zhou and Jiashi Feng and Shuicheng Yan and Xinchao Wang},\n  year = {2022},\n  eprint = {2210.13452},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2210.13452}\n}\n","citationPath":"/content/papers/citations/vision-transformers/caformer.aa84265d674f.bib"},"thumbnail":{"src":"/content/papers/assets/6ccbfbc74d7ee0be5bc2f37dd6cdd936bf9ed5cf9412cdab0f7dcc2c77a6d9ec.jpg","width":1224,"height":1004},"locale":"en","path":"/content/papers/en/vision-transformers/caformer.8024e9840b7d.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"starrelu","text":"StarReLU","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"using-basic-token-mixers","text":"Using Basic Token Mixers","depth":3},{"id":"using-common-token-mixers","text":"Using Common Token Mixers","depth":3},{"id":"ablation-experiments","text":"Ablation Experiments","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/poolformer","relation":"background","topic":"Separating the backbone from token mixing"},{"id":"vision-transformers/convmixer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"classic-cnns/convnext","relation":"comparison","topic":"Convolution for mixing visual patches"}]},{"id":"vision-transformers/poolformer","category":"vision-transformers","legacyPath":"vision-transformers/2111-poolformer","legacySlug":"2111-poolformer","legacyPaths":[],"title":"PoolFormer","subtitle":"You Need a Meta!","author":"Z. Yuan","paperDate":"2021-11-22","publishedOn":"2024-07-10","paperTitle":"MetaFormer Is Actually What You Need for Vision","paperUrl":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","bibliography":{"title":"MetaFormer Is Actually What You Need for Vision","authors":["Weihao Yu","Mi Luo","Pan Zhou","Chenyang Si","Yichen Zhou","Xinchao Wang","Jiashi Feng","Shuicheng Yan"],"url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2022/papers/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.pdf","codeUrl":"https://github.com/sail-sg/poolformer","preprint":{"id":"2111.11418","submittedOn":"2021-11-22","url":"https://arxiv.org/abs/2111.11418"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html"},"citation":"@misc{visiontransformerspoolformer2021,\n  title = {{MetaFormer Is Actually What You Need for Vision}},\n  author = {Weihao Yu and Mi Luo and Pan Zhou and Chenyang Si and Yichen Zhou and Xinchao Wang and Jiashi Feng and Shuicheng Yan},\n  year = {2021},\n  eprint = {2111.11418},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.11418}\n}\n","citationPath":"/content/papers/citations/vision-transformers/poolformer.f2f96629d283.bib"},"thumbnail":{"src":"/content/papers/assets/3cb141e9bb1247410c8214276b21b28ed1d536060d470406f1bfb0c88fe68df2.jpg","width":1498,"height":548},"locale":"en","path":"/content/papers/en/vision-transformers/poolformer.ceadb42af53a.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"input-embedding","text":"Input Embedding","depth":3},{"id":"token-mixer","text":"Token-Mixer","depth":3},{"id":"channel-mlp","text":"Channel MLP","depth":3},{"id":"is-token-mixer-really-the-key","text":"Is Token-Mixer Really the Key?","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"model-configuration","text":"Model Configuration","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet-experiments","text":"ImageNet Experiments","depth":3},{"id":"ablation-experiments","text":"Ablation Experiments","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/mlp-mixer","relation":"background","topic":"Separating the backbone from token mixing"},{"id":"vision-transformers/caformer","relation":"next","topic":"Separating the backbone from token mixing"}]},{"id":"vision-transformers/convmixer","category":"vision-transformers","legacyPath":"vision-transformers/2201-convmixer","legacySlug":"2201-convmixer","legacyPaths":[],"title":"ConvMixer","subtitle":"Convolutional Mixer","author":"Z. Yuan","paperDate":"2022-01-24","publishedOn":"2024-08-25","paperTitle":"Patches Are All You Need?","paperUrl":"https://arxiv.org/abs/2201.09792","bibliography":{"title":"Patches Are All You Need?","authors":["Asher Trockman","J. Zico Kolter"],"url":"https://arxiv.org/abs/2201.09792","pdfUrl":"https://arxiv.org/pdf/2201.09792","codeUrl":"https://github.com/locuslab/convmixer","preprint":{"id":"2201.09792","submittedOn":"2022-01-24","url":"https://arxiv.org/abs/2201.09792"},"publication":{"venue":"Transactions on Machine Learning Research","shortVenue":"TMLR","year":2023,"kind":"journal","url":"https://openreview.net/forum?id=rAnB7JSMXL"},"citation":"@misc{visiontransformersconvmixer2022,\n  title = {{Patches Are All You Need?}},\n  author = {Asher Trockman and J. Zico Kolter},\n  year = {2022},\n  eprint = {2201.09792},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.09792}\n}\n","citationPath":"/content/papers/citations/vision-transformers/convmixer.575f26e378ef.bib"},"thumbnail":{"src":"/content/papers/assets/8285cbfd0782146934fa48acf57a480e4e1296cc35e4e251077fa165f8a659f6.jpg","width":852,"height":704},"locale":"en","path":"/content/papers/en/vision-transformers/convmixer.048c8ad1115f.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"training-strategy","text":"Training Strategy","depth":3},{"id":"model-design-parameters","text":"Model Design Parameters","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"Convolution for mixing visual patches"}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"Making Convolutions Great Again","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"en","path":"/content/papers/en/classic-cnns/convnext.1a8312c7b164.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"redesigning-network-architecture","text":"Redesigning Network Architecture","depth":3},{"id":"introducing-modern-training-techniques","text":"Introducing Modern Training Techniques","depth":3},{"id":"macro-design","text":"Macro Design","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"large-kernel-convolution","text":"Large Kernel Convolution","depth":3},{"id":"micro-design","text":"Micro Design","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"summary-of-model-configurations","text":"Summary of Model Configurations","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"reparameterization/replknet","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]}],"text":"After publishing the first MetaFormer paper, the authors released another paper to demonstrate its performance.\n\n- [**PoolFormer: You Need a Meta!**](/blog/papers/vision-transformers/poolformer)\n\nThis paper builds on that framework and shows us how to assemble models with it. After reading it, we think of it as the MetaFormer user manual. Paper Figure 1 puts the answer up front: identity versus random mixing on the left, convolution versus convolution-plus-attention on the right. Bubble area indicates parameter count, the horizontal axis is MACs, and the vertical axis is ImageNet-1K top-1 accuracy. MACs are not measured runtime.\n\n![Accuracy and MACs of IdentityFormer, RandFormer, ConvFormer, and CAFormer; paper Figure 1](/content/papers/assets/21908bb9378034376f33598804007d75d3f73b7d4759e70fcb4259a8de7a4bc6.jpg \"Paper Figure 1; Yu et al., MetaFormer Baselines for Vision, arXiv:2210.13452v4 / TPAMI 2024\")\n\n## Defining the Problem\n\nThe authors guide us step by step through using MetaFormer. More precisely, PoolFormer showed that simple pooling works within this skeleton. What happens if the mixer does nothing, uses a frozen random matrix, or switches to conventional convolution or attention? How much of the outcome comes from the mixer, and how much from normalization, residuals, activations, and the training recipe? The paper builds baselines to examine these questions. It does not claim that a newly invented mixer solves them all.\n\n## Solving the Problem\n\n### Model Architecture\n\nPaper Figure 2 shows four-stage arrangements on the left and the mixer inside each block on the right. The authors try four token mixers: identity mapping, a frozen random matrix, MobileNetV2-style inverted separable convolution, and the original Transformer's self-attention. The drawing is more useful than memorizing names: check the stages first, then note that the residuals and channel MLP remain.\n\n![Four MetaFormer variants and their stage/block layouts; paper Figure 2](/content/papers/assets/6ccbfbc74d7ee0be5bc2f37dd6cdd936bf9ed5cf9412cdab0f7dcc2c77a6d9ec.jpg \"Paper Figure 2(a)–(h); Yu et al., arXiv:2210.13452v4 / TPAMI 2024\")\n\n1. **IdentityFormer:** all four stages use identity mapping. There is no cross-token mixing in these blocks; the other parts of MetaFormer remain.\n2. **RandFormer:** the first two stages use identity mapping; the last two use a **globally random matrix frozen after initialization**, $W_R\\in\\mathbb{R}^{N\\times N}$, computing $W_RX$. It is neither input-dependent nor a query–key attention map. Parameters and computation grow roughly quadratically with token count, so the authors put it only in the later, shorter-token stages.\n3. **ConvFormer:** every stage uses a MobileNetV2-style separable convolution: pointwise, depthwise, pointwise. Its depthwise kernel is 7 and expansion ratio is 2. This is a convolutional model without self-attention, and the mixer is more than a single depthwise layer.\n4. **CAFormer:** the first two stages use that convolution; the last two use vanilla self-attention. Local, cheaper processing handles many early tokens, while global mixing happens after downsampling.\n\n\n> [!TIP]\n> \n> Read the four variants alongside Figure 2: panels (a)–(d) show the stages and (e)–(h) show individual blocks.\n\n\n### StarReLU\n\nBeyond the basic architecture, the authors introduce **StarReLU**. Let us start with the familiar ReLU and GELU:\n\n$$\n\\operatorname{ReLU}(x)=\\max(0,x).\n$$\n\n$$\n\\operatorname{GELU}(x)=x\\Phi(x)\\approx 0.5x\\left(1+\\tanh\\left(\\sqrt{2/\\pi}(x+0.044715x^3)\\right)\\right).\n$$\n\n$\\Phi$ is the standard Gaussian cumulative distribution function. Under the paper's FLOP counting, ReLU costs 1 FLOP per element and the GELU approximation 14. These are counts for the **activation itself**, not the whole network. To simplify GELU, earlier work tried Squared ReLU:\n\n- [**Primer: Searching for Efficient Transformers for Language Modeling**](https://arxiv.org/abs/2109.08668)\n\nSquared ReLU is straightforward:\n\n$$\n\\operatorname{SquaredReLU}(x)=\\bigl(\\operatorname{ReLU}(x)\\bigr)^2.\n$$\n\nIt costs 2 FLOPs per element in the paper's count. Yet on some image-classification models it does not match GELU, possibly because its output distribution shifts. If $x\\sim\\mathcal N(0,1)$, then\n\n$$\n\\mathbb E[(\\operatorname{ReLU}(x))^2]=0.5,\\qquad \\operatorname{Var}[(\\operatorname{ReLU}(x))^2]=1.25.\n$$\n\nThe authors first derive a standardized form:\n\n$$\n\\operatorname{StarReLU}(x)=\\frac{(\\operatorname{ReLU}(x))^2-\\mathbb E[(\\operatorname{ReLU}(x))^2]}{\\sqrt{\\operatorname{Var}[(\\operatorname{ReLU}(x))^2]}}.\n$$\n\n$$\n\\operatorname{StarReLU}(x)=\\frac{(\\operatorname{ReLU}(x))^2-0.5}{\\sqrt{1.25}}.\n$$\n\n$$\n\\operatorname{StarReLU}(x)\\approx0.8944(\\operatorname{ReLU}(x))^2-0.4472.\n$$\n\nActual network inputs need not follow a standard normal distribution. Thus the scale $s$ and offset $b$ become learnable scalars shared across channels:\n\n$$\n\\operatorname{StarReLU}(x)=s\\bigl(\\operatorname{ReLU}(x)\\bigr)^2+b.\n$$\n\nThe paper estimates **4 FLOPs per element**, about **71% fewer activation FLOPs** than GELU's 14. The name points to its repeated multiplications. Actual speed is another matter: the paper's PyTorch API timing found unfused StarReLU even slower than GELU. Kernel implementation and device matter; 71% is not a whole-network speedup or a latency measurement.\n\n## Discussion\n\n### Using Basic Token Mixers\n\nPaper Table 3 compares identity, random mixing, and PoolFormerV2 on ImageNet-1K at $224\\times224$. **PoolFormerV2** keeps the PoolFormer architecture but applies this paper's updated design/training techniques for comparison. Its numbers should not be silently substituted into the earlier PoolFormer experiment.\n\n\n> [!TIP]\n> \n> PoolFormerV2 retains the pooling mixer from the first PoolFormer; the revised model and training setup explain the V2 label in this table.\n\n\n![IdentityFormer, RandFormer, PoolFormerV2, and RSB-ResNet on ImageNet-1K; paper Table 3](/content/papers/assets/278ad030aa9e31d1d300206b03627493afea9c87ff1e527067b5a80e43ba88e1.jpg \"Paper Table 3; Yu et al., arXiv:2210.13452v4 / TPAMI 2024; underlining marks RandFormer's extra frozen parameters\")\n\nIdentityFormer-S12 reaches **74.6%**, above RSB-ResNet-18 at **70.6%**; S24 reaches **78.2%**, above RSB-ResNet-34 at **75.5%**. Scaling IdentityFormer-M48 to **73.3M** parameters and **11.5G MACs** gives **80.4%**, above RSB-ResNet-50's **79.8%**, though below the table's RSB-ResNet-101 at **81.3%** and ResNet-152 at **81.8%**. The useful claim is that a model with no explicit token mixing still learns substantial representations, not that any model with similar resources is **guaranteed** at least 80%.\n\nRandFormer-S12/M48 report **76.6/81.4%**, up **2.0/1.0 percentage points** over their IdentityFormer counterparts. RandFormer-M36 also reaches **81.2%**, close to RSB-ResNet-101's **81.3%**. S12, however, adds **0.2M frozen matrix parameters** and M48 **0.9M**, with higher MACs. PoolFormerV2-S12/M48 remain higher at **78.0/82.6%**. Frozen random mixing helps, while local pooling's inductive bias is useful in these settings too.\n\n### Using Common Token Mixers\n\nPaper Table 5 groups models by size and separately reports performance at $224$ and, for some, after fine-tuning to $384$. Compare like columns. ConvFormer-B36 obtains **84.8% with 100M parameters and 22.6G MACs**; [**ConvNeXt-L**](/blog/papers/classic-cnns/convnext) gives **84.3% with 198M and 34.4G**. Thus the paper's **0.5-point advantage with 51% of the parameters and 66% of the MACs** is over **ConvNeXt-L**, not ConvNeXt-B. ConvNeXt-B is a different row at **83.8%/89M/15.4G**. ConvFormer-M36 also leads Swin-B/CoAtNet-2 by **1.0/0.4 percentage points**, using **35%/24% fewer parameters** and **17%/18% fewer MACs** in the listed $224$ comparison.\n\n![ImageNet-1K model size, MACs, and top-1 for ConvFormer and CAFormer; paper Table 5](/content/papers/assets/281f9869f8ed29077880ca3e8e6c0be7dd2283622ff2f3d9725ecbd9d35b2507.jpg \"Paper Table 5; Yu et al., arXiv:2210.13452v4 / TPAMI 2024; 224 and 384 resolution are separate columns\")\n\nAt $224\\times224$, CAFormer-B36 reaches **85.5%** with **99M** parameters and **23.2G** MACs, versus **85.3%/218M/42.1G** for MViTv2-L in the same table. This was a record **at the paper's time and under ImageNet-1K supervised training without external data or distillation**, not a current claim across every training regime. After ImageNet-21K pretraining, paper Table 7 reports **87.0%** for ConvFormer-B36 and **87.4%** for CAFormer-B36 at $224$, **2.2/1.9 points** above their ImageNet-1K-only results. Under that same Table 7 setting, ConvFormer-B36 exceeds ConvNeXt-L's **86.8%** by **0.2 points**, with **49% fewer parameters and 34% fewer MACs**. These are separate data conditions from Table 5. The $384$ column also entails extra high-resolution fine-tuning and is not a same-cost comparison with $224$.\n\nCAFormer balances local convolutions while tokens are numerous against attention after downsampling. Table 5 includes different models and training recipes, so it shows this engineering combination is competitive; it does not isolate a single mixer as the sole cause. For large-image detection, the paper also restricts CAFormer's attention to sliding windows to avoid running out of memory. “Global attention in later stages” therefore has a condition when transferred to that task.\n\n### Ablation Experiments\n\nThe baseline in paper Table 8 **uses StarReLU**: ConvFormer-S18 reaches **83.0%**, CAFormer-S18 **83.6%**. Its arrows mean “replace StarReLU **with** ReLU/Squared ReLU/GELU,” not the other direction. ReLU yields **82.1/82.9%**, Squared ReLU **82.6/83.4%**, and GELU **82.7/83.4%**. Thus StarReLU is **0.3/0.2 points** ahead of GELU for these two S18 models. The default uses learnable scale and bias; the table also varies fixed or learned forms.\n\n![StarReLU, ResScale, and block-bias ablations on ImageNet-1K; paper Table 8](/content/papers/assets/0326d533d6284eaff3fbfc11274afda0f425cfb2c67b5d07eb17dd1945559bfe.jpg \"Paper Table 8; Yu et al., arXiv:2210.13452v4 / TPAMI 2024; StarReLU is the baseline\")\n\nResScale performs better here than omitting scaling or switching to LayerScale. Enabling biases within the blocks leaves ConvFormer-S18 almost unchanged and moves CAFormer-S18 from **83.6%** to **83.5%**. These small choices, together with StarReLU, form a better training baseline. They should not be credited entirely to the mixer.\n\n## Conclusion\n\nThe authors use this paper to show what MetaFormer can do. I take four points from their “user manual”:\n\n1. **MetaFormer offers a strong baseline.** Even with identity mapping as its mixer, IdentityFormer-M48 reaches **80.4%** on ImageNet-1K. That is a result for this particular model and training setting.\n2. **It pairs well with different mixers.** A frozen random mixer lifts RandFormer-M48 to **81.4%**, above its IdentityFormer counterpart. That makes the framework a useful starting point when trying a new mixer, although Table 3 still favors PoolFormerV2 at the same scale.\n3. **Familiar mixers can produce strong results.** ConvFormer uses separable convolutions and outperforms the listed ConvNeXt comparison; CAFormer combines them in the early stages with later self-attention and reached **85.5%** at $224\\times224$ under the paper's ImageNet-1K supervised setting, without extra data or distillation. The record is a paper-time claim.\n4. **StarReLU is worth trying.** As a Squared ReLU variant intended to reduce distribution shift, it cuts the paper's activation FLOP count by **71%** relative to GELU and improves the two S18 results in Table 8. That FLOP count is not a measured end-to-end latency gain.\n\nMetaFormer shows considerable room for stronger models. With better mixers or training strategies, we can expect more good results from this framework; the user manual is a place to start building and checking them.\n","imageSizes":{"/content/papers/assets/21908bb9378034376f33598804007d75d3f73b7d4759e70fcb4259a8de7a4bc6.jpg":{"width":1224,"height":548},"/content/papers/assets/6ccbfbc74d7ee0be5bc2f37dd6cdd936bf9ed5cf9412cdab0f7dcc2c77a6d9ec.jpg":{"width":1224,"height":1004},"/content/papers/assets/278ad030aa9e31d1d300206b03627493afea9c87ff1e527067b5a80e43ba88e1.jpg":{"width":1224,"height":1004},"/content/papers/assets/281f9869f8ed29077880ca3e8e6c0be7dd2283622ff2f3d9725ecbd9d35b2507.jpg":{"width":752,"height":1080},"/content/papers/assets/0326d533d6284eaff3fbfc11274afda0f425cfb2c67b5d07eb17dd1945559bfe.jpg":{"width":1224,"height":596}}}
