Back to paper library
arXiv 2022CVPR 2023Vision TransformersPaper notes

FlexiViT

Flexible Patch Sizes

Original paper

FlexiViT: One Model for All Patch Sizes

Paper authors

Lucas Beyer · Pavel Izmailov · Alexander Kolesnikov …

Show all 10 authors

Lucas Beyer · Pavel Izmailov · Alexander Kolesnikov · Mathilde Caron · Simon Kornblith · Xiaohua Zhai · Matthias Minderer · Michael Tschannen · Ibrahim Alabdulmohsin · Filip Pavetic

Publication
CVPR 2023Conference
First submitted to arXiv
arXiv:2212.08013
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{visiontransformersflexivit2022,
  title = {{FlexiViT: One Model for All Patch Sizes}},
  author = {Lucas Beyer and Pavel Izmailov and Alexander Kolesnikov and Mathilde Caron and Simon Kornblith and Xiaohua Zhai and Matthias Minderer and Michael Tschannen and Ibrahim Alabdulmohsin and Filip Pavetic},
  year = {2022},
  eprint = {2212.08013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.08013}
}
On this page

Today, let's take a look at a work from Google Research.

Problem Definition

Paper Figure 1 makes the trade-off plain. Slice the same image into larger patches and there are fewer tokens, making inference faster. Use smaller patches and there are more tokens, potentially improving accuracy at greater cost. Two operating points of the same FlexiViT weights are 84.4%/1.6 ms and 86.1%/13 ms. Those latency examples were measured in the paper's TPUv3 setting, not on a phone or another GPU.

Paper Figure 1; Beyer et al., FlexiViT: One Model for All Patch Sizes, CVPR 2023; timing is a TPUv3 example

The rise of Vision Transformers changed how we represent images: ViT divides an image into fixed-size patches and uses self-attention to process their relationships, instead of relying on the convolutional filters of a traditional CNN. Yet this central design choice is usually fixed once training begins:

Why has no one challenged the patch-size choice?

The authors point out a paradox: patch size changes computational cost and classification performance substantially, yet it usually cannot be adjusted without retraining. So can we

Support several patch sizes with the same set of parameters?

This changes how a model slices the same image. It need not change the image's pixel dimensions too; conflating the two leads to a misreading of the experiments.

Solution

Paper Algorithm 1 looks almost like a joke. Once we have read it, have we finished another paper? Let us go through it line by line: what resize means and how training uses it are the important parts.

Paper Algorithm 1; Beyer et al., CVPR 2023; the illustrative resize operation may use PI-resize according to the setting

Random Patch Size

python
model = ViT(...)
for batch in data:
    ps = np.random.choice([8, 10, ..., 40, 48])
    logits = model(batch["images"], (ps, ps))
    # [...] backprop and optimize as usual

Line 3 draws a patch size for each batch, line 4 passes it to the model, and line 5 performs normal backpropagation. The paper's main setting uses 240×240240\times240 images, sampling approximately uniformly from 8, 10, 12, 15, 16, 20, 24, 30, 40, 48, sizes that divide 240. This is one patch size per batch, not a separate random size for every patch inside an image. Across training steps, the model sees token sequences of different lengths.

For patch size pp, token count is approximately N=(240/p)2N=(240/p)^2. The token-pair part of self-attention grows with N2N^2. Being able to change pp in training and inference creates the practical resource trade-off.

Dynamic Reconfiguration Inside the Model

python
# Algorithm 1's model sketch, not an executable PyTorch class
w = self.param("w_emb", (32, 32, 3, d))
b = self.param("b_emb", d)
w = resize(w, (*patchhw, 3, d))
x = conv(image, w, strides=patchhw) + b

Lines 10–13 of Algorithm 1 keep one underlying set of patch-embedding parameters. The example stores weights of shape 32×32×3×d32\times32\times3\times d, resizes them to p×pp\times p on the fly, and applies a convolution whose kernel and stride are both pp. Position embeddings also start from one underlying 7×7×d7\times7\times d parameter grid and are interpolated to the current token grid. The sizes 32 and 7 are parameter storage choices, not the only patch or grid sizes supported.

The Transformer encoder and remaining weights are shared. Compute cost and behavior at unseen sizes require separate evaluation. The paper trains on a selected set of sizes and evaluates cross-size behavior.

Scalable Positional Encoding

python
pe = self.param("posemb", (7, 7, d))
pe = resize(pe, (*x.shape[1:3], d))
return TransformerEncoder(...)(x + pe)

Changing patch size changes the token grid, so positional embeddings must align with the new positions. The authors use bilinear interpolation for position embeddings. Patch-embedding weights receive a more careful transform, PI-resize. Both operations are called resize in the pseudocode, but they solve different problems: one aligns spatial positions; the other must preserve the scale of the patch projection's dot product.

How to Correctly Adjust the Size?

Let a flattened patch be x∈Rp2x\in\mathbb R^{p^2} and a projection weight be w∈Rp2w\in\mathbb R^{p^2}, giving token e=xTwe=x^Tw. If both are bilinearly enlarged to size 2p2p, duplicated samples make the new dot product much larger. In the paper's simplified example, the old dot product is about one-quarter of the new one. It is not the new product that is one-quarter of the old one. This scale shift makes patch-size changes difficult for an ordinary fixed ViT.

e=⟨x,w⟩.e=\langle x,w\rangle. ⟨x,w⟩≈14⟨resize⁡p2p(x),resize⁡p2p(w)⟩.\langle x,w\rangle\approx\tfrac14\langle \operatorname{resize}^{2p}_{p}(x),\operatorname{resize}^{2p}_{p}(w)\rangle.

Write bilinear resizing as B∈Rp∗2×p2B\in\mathbb R^{p^{*2}\times p^2}, so a resized patch is BxBx. We seek new weights w^\hat w such that

xTw≈(Bx)Tw^.x^Tw\approx(Bx)^T\hat w.

Here bilinear resizing is a linear map BB on the flattened patch:

resize⁡pp∗(o)=Bpp∗vec⁡(o).\operatorname{resize}^{p^*}_{p}(o)=B^{p^*}_{p}\operatorname{vec}(o).

Paper Equation (2) minimizes the expected squared mismatch over a patch distribution XX:

w^∈arg⁡min⁡vEx∼X(xTw−(Bx)Tv)2.\hat w\in\arg\min_v\mathbb E_{x\sim X}\bigl(x^Tw-(Bx)^Tv\bigr)^2.

For upsampling, if BB retains the original vector's information (full column rank), choose

w^=Pw.\hat w=Pw. P=B(BTB)−1=(BT)+.P=B(B^TB)^{-1}=(B^T)^+.

Since BTP=IB^TP=I, this gives

⟨x,w⟩=⟨Bx,w^⟩\langle x,w\rangle=\langle Bx,\hat w\rangle

for every xx. The leading BB in PP cannot be omitted; (BTB)−1(B^TB)^{-1} alone even has the wrong shape.

Paper Figure 4 tests a pretrained ViT-B/8 by resizing both image and weights while keeping sequence length fixed. It compares vanilla bilinear resize, norm and area heuristics, and PI-resize. The green PI-resize line nearly preserves accuracy when scaling up. Ordinary bilinear resizing collapses at larger scales, while downscaling is information-limited for all methods. This isolates the weight transform; it does not say every downstream FlexiViT task has equal accuracy at every patch size.

Paper Figure 4; Beyer et al., CVPR 2023; horizontal axis is image size / patch size, with sequence length held fixed

The Representational World of FlexiViT

FlexiViT aims to maintain useful predictions across patch sizes. A deeper question is:

Do its internal representations remain similar across those sizes?

The authors compare feature maps with CKA, turn similarities into distances with arc-cosine, and visualize them with t-SNE. In paper Figure 6, color marks the layer and point size the token-grid size. Early representations remain relatively close through the MLP before block 6, separate in the middle, and become closer again near the final block. The lower plot's [CLS] clusters stay relatively aligned across grid sizes. This is a similarity visualization. My reading is that the model learns how to take different routes and arrive at the same classification task.

Paper Figure 6; Beyer et al., CVPR 2023; color marks layer and dot size marks grid size

Randomizing patch size is the core training change. Most later transfer experiments also have an important condition: FlexiViT-B is initialized from ViT-B/8 teacher weights. PI-resize maps the teacher patch weights to an underlying 32×3232\times32 grid, and teacher predictions guide distillation. Fixed ViT-B/16 and ViT-B/30 comparators use the same distillation setting. Figure 5 compares initialization and supervision choices and shows that teacher initialization especially helps large patches. We should not assign all downstream performance to patch-size randomization alone.

Discussion

Cross-Task Overview

Space is limited, so let's pick a few figures rather than reprint every experiment. Readers interested in the rest can turn to the original paper. Paper Figure 7 has five groups: SUN397 classification, MSCOCO panoptic segmentation, LVIS open-vocabulary detection, MSCOCO image-text retrieval, and Cityscapes semantic segmentation. Each /30 or /16 denotes the patch size used for transfer and evaluation. The green bars use one pretrained FlexiViT-B, compared with orange ViT-B/30 and purple ViT-B/16 baselines at their corresponding sizes. Most are close; some are higher and some lower. On /30 LVIS and retrieval, green is slightly below the fixed counterpart; on /16 panoptic segmentation it is slightly above. The classification top-1, PQ, AP, R@1 and mIoU bars have different metrics and scales; they cannot be combined into one percentage-point gap. There is no result here saying every task and size improves.

Paper Figure 7; Beyer et al., CVPR 2023; each panel has a different metric and vertical scale

One pretrained visual backbone transferring to several tasks and sizes is valuable. Each task still needs its own head, text tower, or fine-tuning procedure. Thus “one model replacing many” refers to shared visual pretraining weights, not a single complete deployment artifact for every application.

Fast Transfer and Inference Flexibility

In the left plot of paper Figure 8, FlexiViT-B and fixed ViT-B/30 are fine-tuned on ImageNet-1K at only an 8×88\times8 token grid, then evaluated at other grids. FlexiViT gives 81.8% at 8×88\times8 and 85.3% at 24×2424\times24, a gain of 3.5 percentage points. The fixed ViT curve first rises a little, then declines quickly as the grid grows denser. The right plot fine-tunes the same pretrained FlexiViT separately at 8×88\times8, 15×1515\times15, and 30×3030\times30. All can be evaluated across grids, though the 8×88\times8 and 15×1515\times15 curves dip slightly at the densest grid rather than rising without limit.

Paper Figure 8; Beyer et al., CVPR 2023; horizontal axis is token-grid size, not patch size; larger grids cost more at inference

This offers cheap fine-tuning followed by a choice to use more tokens at inference. It needs no additional fine-tuning, but inference cost does rise. A/B tests and tiered deployments may benefit; latency and memory still need measurement on the target system.

Open-Vocabulary Object Detection

Paper Figure 11 introduces flexible-patch training to OWL-ViT and evaluates zero-shot open-vocabulary detection AP on LVIS. The green Flexi OWL-ViT line remains useful from /48 to /12; at /24 and /30 it exceeds the fixed /30 model, and at /12 it still reaches about 26% AP. Around /16, though, the fixed /16 model's orange dashed curve slightly exceeds it. The green model does not win at every point. The horizontal axis is patch size: moving right means smaller patches and more tokens.

Paper Figure 11; Beyer et al., CVPR 2023; AP is plotted against inference patch size

The authors also find on more detection datasets that the best patch size need not be the smallest. This is relevant when objects appear at many scales. The choice still requires a validation set and resource budget.

Time Is Compute

Paper Figure 12 asks whether flexible-patch training can save pretraining compute even if the final target is a fixed ViT-B/16. Orange points are fixed /16 training; blue points use flexible-patch schedules and end at /16. All are evaluated at /16. The vertical axis is ImageNet-21K validation precision and the horizontal axis is TPU core-hours. At lower compute budgets, blue points often exceed nearby orange points, but not every point wins; around 400 hours, the displayed gap is less than one percentage point. The authors' appendix also describes the saving as real but limited. The chart does not support a constant “one percentage point at 400 hours” claim.

Paper Figure 12; Beyer et al., CVPR 2023; points reflect different training lengths and curricula

One possible curriculum starts with larger patches, reducing tokens per step, then shifts probability toward the target /16. This is a training configuration tested by the authors, not a plug-and-play speed guarantee for every dataset.

Conclusion

FlexiViT gives us a simple way to use one visual backbone across different compute and resolution needs. It does not add an elaborate new architecture. Instead, it returns to ViT's basic design choice, patchification, and treats patch size as a way to balance computation and the visual detail available to the model.

Random patch-size training, PI-resize of embedding weights, and teacher initialization and distillation help that shared backbone transfer across classification, segmentation, open-vocabulary detection, and image–text retrieval. Figures 7–8 show that the results generally remain close to fixed-patch alternatives; Figures 11–12 show why the best operating point still depends on the task and compute budget.

FlexiViT invites us to rethink the role of patchification. Perhaps we will see more ideas here, from token encoding and structural compression to multiscale fusion and data-driven scheduling. Smaller patches may buy accuracy, but they also bring more tokens, memory, and inference time; that trade-off is what makes a flexible choice useful.