FlexiViT: One Model for All Patch Sizes
- Publication
- CVPR 2023Conference
- First submitted to arXiv
- arXiv:2212.08013
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{visiontransformersflexivit2022,
title = {{FlexiViT: One Model for All Patch Sizes}},
author = {Lucas Beyer and Pavel Izmailov and Alexander Kolesnikov and Mathilde Caron and Simon Kornblith and Xiaohua Zhai and Matthias Minderer and Michael Tschannen and Ibrahim Alabdulmohsin and Filip Pavetic},
year = {2022},
eprint = {2212.08013},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2212.08013}
}
Today, let's take a look at a work from Google Research.
Problem Definition
Paper Figure 1 makes the trade-off plain. Slice the same image into larger patches and there are fewer tokens, making inference faster. Use smaller patches and there are more tokens, potentially improving accuracy at greater cost. Two operating points of the same FlexiViT weights are 84.4%/1.6 ms and 86.1%/13 ms. Those latency examples were measured in the paper's TPUv3 setting, not on a phone or another GPU.
The rise of Vision Transformers changed how we represent images: ViT divides an image into fixed-size patches and uses self-attention to process their relationships, instead of relying on the convolutional filters of a traditional CNN. Yet this central design choice is usually fixed once training begins:
Why has no one challenged the patch-size choice?
The authors point out a paradox: patch size changes computational cost and classification performance substantially, yet it usually cannot be adjusted without retraining. So can we
Support several patch sizes with the same set of parameters?
This changes how a model slices the same image. It need not change the image's pixel dimensions too; conflating the two leads to a misreading of the experiments.
Solution
Paper Algorithm 1 looks almost like a joke. Once we have read it, have we finished another paper? Let us go through it line by line: what resize means and how training uses it are the important parts.
Random Patch Size
model = ViT(...)
for batch in data:
ps = np.random.choice([8, 10, ..., 40, 48])
logits = model(batch["images"], (ps, ps))
# [...] backprop and optimize as usual
Line 3 draws a patch size for each batch, line 4 passes it to the model, and line 5 performs normal backpropagation. The paper's main setting uses images, sampling approximately uniformly from 8, 10, 12, 15, 16, 20, 24, 30, 40, 48, sizes that divide 240. This is one patch size per batch, not a separate random size for every patch inside an image. Across training steps, the model sees token sequences of different lengths.
For patch size , token count is approximately . The token-pair part of self-attention grows with . Being able to change in training and inference creates the practical resource trade-off.
Dynamic Reconfiguration Inside the Model
# Algorithm 1's model sketch, not an executable PyTorch class
w = self.param("w_emb", (32, 32, 3, d))
b = self.param("b_emb", d)
w = resize(w, (*patchhw, 3, d))
x = conv(image, w, strides=patchhw) + b
Lines 10–13 of Algorithm 1 keep one underlying set of patch-embedding parameters. The example stores weights of shape , resizes them to on the fly, and applies a convolution whose kernel and stride are both . Position embeddings also start from one underlying parameter grid and are interpolated to the current token grid. The sizes 32 and 7 are parameter storage choices, not the only patch or grid sizes supported.
The Transformer encoder and remaining weights are shared. Compute cost and behavior at unseen sizes require separate evaluation. The paper trains on a selected set of sizes and evaluates cross-size behavior.
Scalable Positional Encoding
pe = self.param("posemb", (7, 7, d))
pe = resize(pe, (*x.shape[1:3], d))
return TransformerEncoder(...)(x + pe)
Changing patch size changes the token grid, so positional embeddings must align with the new positions. The authors use bilinear interpolation for position embeddings. Patch-embedding weights receive a more careful transform, PI-resize. Both operations are called resize in the pseudocode, but they solve different problems: one aligns spatial positions; the other must preserve the scale of the patch projection's dot product.
How to Correctly Adjust the Size?
Let a flattened patch be and a projection weight be , giving token . If both are bilinearly enlarged to size , duplicated samples make the new dot product much larger. In the paper's simplified example, the old dot product is about one-quarter of the new one. It is not the new product that is one-quarter of the old one. This scale shift makes patch-size changes difficult for an ordinary fixed ViT.
Write bilinear resizing as , so a resized patch is . We seek new weights such that
Here bilinear resizing is a linear map on the flattened patch:
Paper Equation (2) minimizes the expected squared mismatch over a patch distribution :
For upsampling, if retains the original vector's information (full column rank), choose
Since , this gives
for every . The leading in cannot be omitted; alone even has the wrong shape.
Paper Figure 4 tests a pretrained ViT-B/8 by resizing both image and weights while keeping sequence length fixed. It compares vanilla bilinear resize, norm and area heuristics, and PI-resize. The green PI-resize line nearly preserves accuracy when scaling up. Ordinary bilinear resizing collapses at larger scales, while downscaling is information-limited for all methods. This isolates the weight transform; it does not say every downstream FlexiViT task has equal accuracy at every patch size.
The Representational World of FlexiViT
FlexiViT aims to maintain useful predictions across patch sizes. A deeper question is:
Do its internal representations remain similar across those sizes?
The authors compare feature maps with CKA, turn similarities into distances with arc-cosine, and visualize them with t-SNE. In paper Figure 6, color marks the layer and point size the token-grid size. Early representations remain relatively close through the MLP before block 6, separate in the middle, and become closer again near the final block. The lower plot's [CLS] clusters stay relatively aligned across grid sizes. This is a similarity visualization. My reading is that the model learns how to take different routes and arrive at the same classification task.
Randomizing patch size is the core training change. Most later transfer experiments also have an important condition: FlexiViT-B is initialized from ViT-B/8 teacher weights. PI-resize maps the teacher patch weights to an underlying grid, and teacher predictions guide distillation. Fixed ViT-B/16 and ViT-B/30 comparators use the same distillation setting. Figure 5 compares initialization and supervision choices and shows that teacher initialization especially helps large patches. We should not assign all downstream performance to patch-size randomization alone.
Discussion
Cross-Task Overview
Space is limited, so let's pick a few figures rather than reprint every experiment. Readers interested in the rest can turn to the original paper. Paper Figure 7 has five groups: SUN397 classification, MSCOCO panoptic segmentation, LVIS open-vocabulary detection, MSCOCO image-text retrieval, and Cityscapes semantic segmentation. Each /30 or /16 denotes the patch size used for transfer and evaluation. The green bars use one pretrained FlexiViT-B, compared with orange ViT-B/30 and purple ViT-B/16 baselines at their corresponding sizes. Most are close; some are higher and some lower. On /30 LVIS and retrieval, green is slightly below the fixed counterpart; on /16 panoptic segmentation it is slightly above. The classification top-1, PQ, AP, R@1 and mIoU bars have different metrics and scales; they cannot be combined into one percentage-point gap. There is no result here saying every task and size improves.
One pretrained visual backbone transferring to several tasks and sizes is valuable. Each task still needs its own head, text tower, or fine-tuning procedure. Thus “one model replacing many” refers to shared visual pretraining weights, not a single complete deployment artifact for every application.
Fast Transfer and Inference Flexibility
In the left plot of paper Figure 8, FlexiViT-B and fixed ViT-B/30 are fine-tuned on ImageNet-1K at only an token grid, then evaluated at other grids. FlexiViT gives 81.8% at and 85.3% at , a gain of 3.5 percentage points. The fixed ViT curve first rises a little, then declines quickly as the grid grows denser. The right plot fine-tunes the same pretrained FlexiViT separately at , , and . All can be evaluated across grids, though the and curves dip slightly at the densest grid rather than rising without limit.
This offers cheap fine-tuning followed by a choice to use more tokens at inference. It needs no additional fine-tuning, but inference cost does rise. A/B tests and tiered deployments may benefit; latency and memory still need measurement on the target system.
Open-Vocabulary Object Detection
Paper Figure 11 introduces flexible-patch training to OWL-ViT and evaluates zero-shot open-vocabulary detection AP on LVIS. The green Flexi OWL-ViT line remains useful from /48 to /12; at /24 and /30 it exceeds the fixed /30 model, and at /12 it still reaches about 26% AP. Around /16, though, the fixed /16 model's orange dashed curve slightly exceeds it. The green model does not win at every point. The horizontal axis is patch size: moving right means smaller patches and more tokens.
The authors also find on more detection datasets that the best patch size need not be the smallest. This is relevant when objects appear at many scales. The choice still requires a validation set and resource budget.
Time Is Compute
Paper Figure 12 asks whether flexible-patch training can save pretraining compute even if the final target is a fixed ViT-B/16. Orange points are fixed /16 training; blue points use flexible-patch schedules and end at /16. All are evaluated at /16. The vertical axis is ImageNet-21K validation precision and the horizontal axis is TPU core-hours. At lower compute budgets, blue points often exceed nearby orange points, but not every point wins; around 400 hours, the displayed gap is less than one percentage point. The authors' appendix also describes the saving as real but limited. The chart does not support a constant “one percentage point at 400 hours” claim.
One possible curriculum starts with larger patches, reducing tokens per step, then shifts probability toward the target /16. This is a training configuration tested by the authors, not a plug-and-play speed guarantee for every dataset.
Conclusion
FlexiViT gives us a simple way to use one visual backbone across different compute and resolution needs. It does not add an elaborate new architecture. Instead, it returns to ViT's basic design choice, patchification, and treats patch size as a way to balance computation and the visual detail available to the model.
Random patch-size training, PI-resize of embedding weights, and teacher initialization and distillation help that shared backbone transfer across classification, segmentation, open-vocabulary detection, and image–text retrieval. Figures 7–8 show that the results generally remain close to fixed-patch alternatives; Figures 11–12 show why the best operating point still depends on the task and compute budget.
FlexiViT invites us to rethink the role of patchification. Perhaps we will see more ideas here, from token encoding and structural compression to multiscale fusion and data-driven scheduling. Smaller patches may buy accuracy, but they also bring more tokens, memory, and inference time; that trade-off is what makes a flexible choice useful.