MLP-Mixer: An all-MLP Architecture for Vision
- Publication
- NeurIPS 2021Conference
- First submitted to arXiv
- arXiv:2105.01601
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{visiontransformersmlpmixer2021,
title = {{MLP-Mixer: An all-MLP Architecture for Vision}},
author = {Ilya Tolstikhin and Neil Houlsby and Alexander Kolesnikov and Lucas Beyer and Xiaohua Zhai and Thomas Unterthiner and Jessica Yung and Andreas Steiner and Daniel Keysers and Jakob Uszkoreit and Mario Lucic and Alexey Dosovitskiy},
year = {2021},
eprint = {2105.01601},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2105.01601}
}
After Transformers were introduced into the field of computer vision, many studies began exploring ways to make the exchange of information between patches more efficient.
Thus, a new term emerged: "Token-Mixer."
As the name implies, the purpose of a Token-Mixer is to mix information between different patches to enhance the performance of the model.
The traditional self-attention mechanism is one such implementation of a Token-Mixer.
However, due to the high computational complexity of the original attention mechanism, the academic community began searching for alternatives.
MLP-Mixer is one of these alternatives.
Problem Definition
The authors believe: the complexity of self-attention is too high, so let's just get rid of it!
But what should we do after getting rid of it?
Hey! Look, there's a simple fully connected layer by the roadside! Let's use it!
And then the paper ends here. (Not really!)
Solution
Model Architecture
In this paper, the authors propose using two stages of fully connected layers, or MLPs (Multi-Layer Perceptrons), to exchange information.
The image above might be hard to understand, but don't worry, let's break it down step by step:
-
Input an image of size 3 x 224 x 224 and divide it into 16 x 16 patches, with a dimension assumed to be 512.
- Dimension change: [B x 3 x 224 x 224] -> [B x 512 x 14 x 14].
-
Flatten the patches into a sequence. This borrows the layout of a Transformer input; the sequence is not passed through a Transformer.
- Dimension change: [B x 512 x 14 x 14] -> [B x 512 x 196].
-
First stage: Exchange information between patches.
import torch import torch.nn as nn # Input: [batch, channels, tokens]; an illustrative 512-channel block. x = torch.randn(2, 512, 196) norm_tokens = nn.LayerNorm(512) patch_mixer = nn.Sequential( nn.Linear(196, 256), nn.GELU(), nn.Linear(256, 196) ) y = norm_tokens(x.transpose(1, 2)).transpose(1, 2) x = x + patch_mixer(y) -
Second stage: Exchange information between channels.
# Continue from the token-mixing result above. norm_channels = nn.LayerNorm(512) channel_mixer = nn.Sequential( nn.Linear(512, 2048), nn.GELU(), nn.Linear(2048, 512) ) y = norm_channels(x.transpose(1, 2)) x = x + channel_mixer(y).transpose(1, 2) assert x.shape == (2, 512, 196)
This completes one operation of the MLP-Mixer.
The second stage resembles the feed-forward network after attention in a ViT block: it mixes channels within each token, but is not self-attention itself. Each mixer uses two linear layers with GELU, pre-LayerNorm and a residual. The complete model ends with global average pooling and a classifier; this illustrative example omits dropout, patch embedding and the classifier.
The first stage uses the same learned weights across channels to mix all 196 patches, rather than computing an input-dependent QK attention map. Its fixed-position mixing makes separate position embeddings unnecessary. With the same patch size, a 196→256→196 MLP is tied to that token count; fine-tuning at a higher resolution requires adjusting the mixing weights.
Discussion
Performance on ImageNet
- The "ImNet" and "ReaL" columns represent the results on the original ImageNet validation set and the cleaned ReaL label set, respectively.
- Avg 5 indicates the average performance across five downstream tasks (ImageNet, CIFAR-10, CIFAR-100, Pets, Flowers).
When pre-trained on ImageNet-21k with additional regularization and fine-tuned at resolution 448, Mixer-L/16 achieves 84.15% top-1 accuracy on ImageNet. This is strong performance, but slightly below several comparison models in the table. Without regularization, Mixer tends to overfit, consistent with observations for ViT.
When training MLP-Mixer from scratch on ImageNet, Mixer-B/16 scores 76.4% at resolution 224, similar to a standard ResNet50 but lower than other state-of-the-art CNN/hybrid models like BoTNet (84.7%) and NFNet (86.5%).
As the size of the upstream dataset increases, MLP-Mixer's performance improves significantly. The right-hand plot above evaluates the fixed pre-trained representations with a linear 5-shot classifier, not full fine-tuning.
With JFT-300M pretraining and fine-tuning at 448 resolution, Mixer-H/14 reaches 87.94% top-1. Table values for BiT-R152x4 and ViT-H/14 are 87.54% and 88.55%: differences of +0.40 and −0.61 percentage points. TPU-v3 throughput per core is 40, 26 and 15 img/s respectively, making Mixer about 2.67× ViT and 1.54× BiT in this benchmark, not universally across hardware.
In terms of precision-computation trade-offs, Mixer is competitive with more traditional neural network architectures. There is a clear correlation between total pre-training cost and downstream accuracy across architecture categories.
Visualization Analysis
It is generally observed that the first layer of CNNs tends to learn Gabor-like detectors, which act on local regions of the image pixels.
In contrast, MLP-Mixer allows global information exchange between tokens in the MLP, raising the question:
- Does it process information in a similar way?
The figure visualizes hidden-unit weights of the first three token-mixing MLPs in Mixer-B/16 trained on JFT-300M. Each unit’s 196 weights are reshaped into a 14×14 grid. These are not activation maps produced by a particular input image.
In the image above, from left to right, are the first, second, and third layers of MLP-Mixer. We can see:
- Some learned features act on the entire image, while others act on smaller regions.
- Deeper layers seem to have less distinguishable structures.
- Similar to CNNs, there are many feature detectors with opposite phases.
Conclusion
Losing less counts as a gain!
Replacing the original self-attention with fixed-width token mixing can save a great deal of computation. Even if its results are a little worse, there is still a market for it!