{"notes":[{"id":"classic-cnns/convnets-match-vits","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ConvNets Match ViTs","subtitle":"The Bitter Lesson","author":"Z. Yuan","paperDate":"2023-10-25","publishedOn":"2025-11-17","paperTitle":"ConvNets Match Vision Transformers at Scale","paperUrl":"https://arxiv.org/abs/2310.16764","bibliography":{"title":"ConvNets Match Vision Transformers at Scale","authors":["Samuel L. Smith","Andrew Brock","Leonard Berrada","Soham De"],"url":"https://arxiv.org/abs/2310.16764","pdfUrl":"https://arxiv.org/pdf/2310.16764v1","codeUrl":null,"preprint":{"id":"2310.16764","submittedOn":"2023-10-25","url":"https://arxiv.org/abs/2310.16764"},"publication":null,"citation":"@misc{classiccnnsconvnetsmatchvits2023,\n  title = {{ConvNets Match Vision Transformers at Scale}},\n  author = {Samuel L. Smith and Andrew Brock and Leonard Berrada and Soham De},\n  year = {2023},\n  eprint = {2310.16764},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2310.16764}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnets-match-vits.0657c1e27561.bib"},"thumbnail":{"src":"/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png","width":1830,"height":828},"locale":"en","path":"/content/papers/en/classic-cnns/convnets-match-vits.80493f9733ae.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"scaling-laws-for-pretraining-loss","text":"Scaling Laws for Pretraining Loss","depth":3},{"id":"learning-rate","text":"Learning Rate","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet-fine-tuning","text":"ImageNet Fine-Tuning","depth":3},{"id":"lowest-loss-does-not-always-give-the-best-fine-tuning","text":"Lowest Loss Does Not Always Give the Best Fine-Tuning","depth":3},{"id":"scope-of-the-comparison","text":"Scope of the Comparison","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/vit","relation":"referenced","topic":""},{"id":"classic-cnns/nfnet","relation":"referenced","topic":""},{"id":"transformers/scaling_laws","relation":"referenced","topic":""}]},{"id":"vision-transformers/vit","category":"vision-transformers","legacyPath":"vision-transformers/2010-vit","legacySlug":"2010-vit","legacyPaths":[],"title":"ViT","subtitle":"Pioneering a New World","author":"Z. Yuan","paperDate":"2020-10-22","publishedOn":"2024-07-01","paperTitle":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","paperUrl":"https://arxiv.org/abs/2010.11929","bibliography":{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","authors":["Alexey Dosovitskiy","Lucas Beyer","Alexander Kolesnikov","Dirk Weissenborn","Xiaohua Zhai","Thomas Unterthiner","Mostafa Dehghani","Matthias Minderer","Georg Heigold","Sylvain Gelly","Jakob Uszkoreit","Neil Houlsby"],"url":"https://arxiv.org/abs/2010.11929","pdfUrl":"https://arxiv.org/pdf/2010.11929","codeUrl":null,"preprint":{"id":"2010.11929","submittedOn":"2020-10-22","url":"https://arxiv.org/abs/2010.11929"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://openreview.net/pdf?id=YicbFdNTTy"},"citation":"@misc{visiontransformersvit2020,\n  title = {{An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale}},\n  author = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},\n  year = {2020},\n  eprint = {2010.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.11929}\n}\n","citationPath":"/content/papers/citations/vision-transformers/vit.8a478b78ec38.bib"},"thumbnail":{"src":"/content/papers/assets/82fb5d2f7ce623daf8d9ce146f096d22f2fb17d2532c5c4d613206f0424904aa.jpg","width":1472,"height":772},"locale":"en","path":"/content/papers/en/vision-transformers/vit.ae37d64b0944.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"patchify","text":"Patchify","depth":3},{"id":"and-then","text":"And Then?","depth":3},{"id":"lack-of-inductive-bias","text":"Lack of Inductive Bias","depth":3},{"id":"training-data-must-be-large","text":"Training Data Must Be Large","depth":3},{"id":"can-be-even-larger","text":"Can Be Even Larger","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"what-does-vit-see","text":"What Does ViT See?","depth":3},{"id":"predict-with-cls-or-gap","text":"Predict with [CLS] or GAP?","depth":3},{"id":"self-supervised-training","text":"Self-Supervised Training","depth":3},{"id":"other-considerations","text":"Other Considerations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/deit","relation":"next","topic":"Learning visual representations with Transformers"},{"id":"vision-transformers/pvt","relation":"comparison","topic":"Multiscale visual Transformers"},{"id":"vision-transformers/swin-transformer","relation":"comparison","topic":"Multiscale visual Transformers"}]},{"id":"classic-cnns/nfnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NFNet","subtitle":"Life Without BN","author":"Z. Yuan","paperDate":"2021-02-11","publishedOn":"2025-01-21","world":"external/nfnet","paperTitle":"High-Performance Large-Scale Image Recognition Without Normalization","paperUrl":"https://arxiv.org/abs/2102.06171","bibliography":{"title":"High-Performance Large-Scale Image Recognition Without Normalization","authors":["Andrew Brock","Soham De","Samuel L. Smith","Karen Simonyan"],"url":"https://arxiv.org/abs/2102.06171","pdfUrl":"https://arxiv.org/pdf/2102.06171v1","codeUrl":null,"preprint":{"id":"2102.06171","submittedOn":"2021-02-11","url":"https://arxiv.org/abs/2102.06171"},"publication":{"venue":"ICML","year":2021,"kind":"conference","url":"https://proceedings.mlr.press/v139/brock21a.html"},"citation":"@misc{classiccnnsnfnet2021,\n  title = {{High-Performance Large-Scale Image Recognition Without Normalization}},\n  author = {Andrew Brock and Soham De and Samuel L. Smith and Karen Simonyan},\n  year = {2021},\n  eprint = {2102.06171},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.06171}\n}\n","citationPath":"/content/papers/citations/classic-cnns/nfnet.91baba59df96.bib"},"thumbnail":{"src":"/content/papers/assets/7552bff30433f4f33bec5edb2421acd94d61d452888dde6df17557778e8fb1c6.png","width":1004,"height":783},"locale":"en","path":"/content/papers/en/classic-cnns/nfnet.6c1bcaebe98e.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"what-batchnorm-actually-does-for-us","text":"What BatchNorm Actually Does for Us","depth":3},{"id":"what-mean-shift-looks-like","text":"What Mean Shift Looks Like","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"first-finishing-nf-resnet","text":"First, Finishing NF-ResNet","depth":3},{"id":"adaptive-gradient-clipping","text":"Adaptive Gradient Clipping","depth":3},{"id":"the-nfnet-architecture","text":"The NFNet Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"what-each-change-contributes","text":"What Each Change Contributes","depth":3},{"id":"is-removing-bn-really-faster","text":"Is Removing BN Really Faster?","depth":3},{"id":"against-other-models","text":"Against Other Models","depth":3},{"id":"fine-tuning-after-pretraining","text":"Fine-Tuning After Pretraining","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/batchnorm","relation":"referenced","topic":""}]},{"id":"transformers/scaling_laws","category":"transformers","legacyPath":"transformers/2001-scaling_laws","legacySlug":"2001-scaling_laws","legacyPaths":[],"title":"Scaling Laws","subtitle":"Scaling Laws for Neural Language Models","author":"Z. Yuan","paperDate":"2020-01-23","publishedOn":"2024-05-24","paperTitle":"Scaling Laws for Neural Language Models","paperUrl":"https://arxiv.org/abs/2001.08361","bibliography":{"title":"Scaling Laws for Neural Language Models","authors":["Jared Kaplan","Sam McCandlish","Tom Henighan","Tom B. Brown","Benjamin Chess","Rewon Child","Scott Gray","Alec Radford","Jeffrey Wu","Dario Amodei"],"url":"https://arxiv.org/abs/2001.08361","pdfUrl":"https://arxiv.org/pdf/2001.08361","codeUrl":null,"preprint":{"id":"2001.08361","submittedOn":"2020-01-23","url":"https://arxiv.org/abs/2001.08361"},"publication":null,"citation":"@misc{transformersscalinglaws2020,\n  title = {{Scaling Laws for Neural Language Models}},\n  author = {Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei},\n  year = {2020},\n  eprint = {2001.08361},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2001.08361}\n}\n","citationPath":"/content/papers/citations/transformers/scaling_laws.a1ab0069ac66.bib"},"thumbnail":{"src":"/content/papers/assets/1a680a937f0c09dd160f2661fc04c4598f9bec153e122a7b92c264ef213558c1.jpg","width":1696,"height":544},"locale":"en","path":"/content/papers/en/transformers/scaling_laws.f5c1f7d57894.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"training-process","text":"Training Process","depth":3},{"id":"dataset","text":"Dataset","depth":3},{"id":"variation-factors","text":"Variation Factors","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"size-matters-most","text":"Size Matters Most","depth":3},{"id":"power-laws","text":"Power Laws","depth":3},{"id":"the-universality-of-overfitting","text":"The Universality of Overfitting","depth":3},{"id":"test-performance-vs-downstream-performance","text":"Test Performance vs. Downstream Performance","depth":3},{"id":"sample-efficiency","text":"Sample Efficiency","depth":3},{"id":"convergence-is-inefficient","text":"Convergence is Inefficient","depth":3},{"id":"optimal-batch-size","text":"Optimal Batch Size","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/gpt_3","relation":"background","topic":"Trading model size against training data"},{"id":"transformers/chinchilla","relation":"next","topic":"Trading model size against training data"}]}],"text":"“Convolutional networks do well on small and medium datasets, but fall behind Vision Transformers once the data reaches web scale.”\n\nThe paper opens with this view, which the authors say many researchers hold. ViTs have grown beyond a billion parameters and been pretrained on billions of images, while few convolutional networks have been trained at the same scale.\n\n\n> [!TIP]\n> \n> For how ViT splits images into patches and feeds them into a Transformer, see:\n> \n> - **[[20.10] ViT: Pioneering a New World](/blog/papers/vision-transformers/vit)**\n\n\nThis paper comes from Google DeepMind, by Samuel L. Smith, Andrew Brock, Leonard Berrada, and Soham De. Andrew Brock is NFNet's first author. The main text is just four pages, and the authors themselves call it a note in the acknowledgments. It asks one question: if a convolutional network and a ViT receive similar pretraining compute, does the ViT still win?\n\n## Defining the Problem\n\nThe authors argue that evidence for this belief is thin.\n\nFirst, many ViT papers compare against the original ResNet, effectively comparing a new training recipe with an old baseline.\n\nSecond, the compute gap is huge. The strongest ViTs use over 500,000 TPU-v3 core hours for pretraining, far beyond what convolutional networks have received.\n\nSo the authors change neither the architecture nor the training pipeline. They tune simple hyperparameters such as learning rate and epoch count, train NFNet at the same scale, and see what happens. NFNet is a pure convolutional network from 2021, around the time of the first ViT papers. The authors describe it as the last convolutional network to set a new ImageNet state of the art.\n\n\n> [!TIP]\n> \n> NFNet removes BatchNorm and uses adaptive gradient clipping to stabilize large-batch training. See:\n> \n> - **[[21.02] NFNet: Life Without BN](/blog/papers/classic-cnns/nfnet)**\n\n\n## Solving the Problem\n\n### Experimental Setup\n\nPretraining uses JFT-4B: roughly 4 billion labeled images in 30,000 classes. Near-duplicates of the ImageNet training and validation images are removed first. This dataset is private.\n\nThe models vary in depth and width within the NFNet family. F1 is twice as deep as F0, F3 is twice as deep as F1, and F7 is twice as deep as F3. F3+ and F7+ keep the same depth while increasing width.\n\nEach model is trained several times for between 0.25 and 8 epochs, with cosine learning-rate decay. The optimizer is momentum SGD with NFNet's original adaptive gradient clipping (AGC), at batch size 4096 and training resolution 224×224.\n\nCompute is measured in TPU-v4 core hours, ranging from 0.4k to 110k. A TPU-v4 core has about twice the theoretical compute of a TPU-v3 core and similar memory, so comparisons with the TPU-v3 numbers in ViT papers need conversion, as discussed below.\n\n### Scaling Laws for Pretraining Loss\n\n[![Figure 2 from the paper: a log-log plot with pretraining TPU-v4 core hours on the horizontal axis, about 10³ to 10⁵, and validation loss on the JFT-4B held-out set on the vertical axis, about 2.4 to 3.0. Six curves correspond to F0, F1, F3, F3+, F7, and F7+; points on each curve use different epoch counts. The curves roughly follow one line from upper left to lower right, with some irregularity in the middle of the F7+ curve.](/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png)](/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png)\n\n*Figure 2 from Smith et al., taken from [**PDF page 3**](https://arxiv.org/pdf/2310.16764v1#page=3). The original caption is omitted.*\n\nEach curve represents one model, with points for different epoch counts. Both axes are logarithmic, and the lower envelope of the curves is roughly straight: validation loss follows a power law in compute. This is what language-model scaling laws look like, and NFNet follows the same pattern.\n\nFor each compute budget, one model size and epoch count minimize loss. The authors find that both grow together. Their rule of thumb is to scale model size and training epochs at the same rate, consistent with Chinchilla's conclusion for language models. Above roughly 5k TPU-v4 core hours, the optimal epoch count exceeds 1.\n\nThe F7+ curve is uneven. The authors explain that when training is interrupted and restarted, the input pipeline does not guarantee that every image is seen exactly once per epoch. After several restarts, some examples are seen less often.\n\n\n> [!TIP]\n> \n> For language-model scaling laws and Chinchilla, see our notes:\n> \n> - **[[20.01] Scaling Laws: Scaling Laws for Neural Language Models](/blog/papers/transformers/scaling_laws)**\n> - **[[22.03] Chinchilla: Chinchilla's Gaze](/blog/papers/transformers/chinchilla)**\n\n\n### Learning Rate\n\n[![Figure 3 from the paper: the horizontal axis is training epochs from 0.25 to 8 on a log scale, and the vertical axis is the optimal learning rate from 0.4 to 1.6. F0, F3, and F7+ all stay at 1.6 from 0.25 to 1 epoch. The dashed F7+ line drops to 0.8 at 2 epochs and 0.4 at 4; F3 is at 0.8 from 2 to 4 epochs and 0.4 at 8. F0 remains at 1.6 through 4 epochs and falls to 0.8 at 8.](/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png)](/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png)\n\n*Figure 3 from Smith et al., taken from [**PDF page 3**](https://arxiv.org/pdf/2310.16764v1#page=3). The original caption is omitted.*\n\nThe learning rate is searched on a logarithmic grid spaced by factors of 2. Up to 1 epoch, all three models prefer 1.6. With more epochs, the optimal rate begins to decrease, and larger models reach that point earlier.\n\nThe practical value lies in that last observation. Since the optimal learning rate decreases slowly and monotonically with model size and epoch count, the authors say it can usually be found in about two trials.\n\n## Discussion\n\n### ImageNet Fine-Tuning\n\nThe pretrained models are fine-tuned on ImageNet for 50 epochs with SAM, stochastic depth, and dropout. Training resolution is 384×384; evaluation resolution is 480×480.\n\n[![Figure 1 from the paper: a log-log plot with pretraining TPU-v4 core hours on the horizontal axis, about 10³ to 10⁵, and ImageNet top-1 error on the vertical axis, about 9.5% to 14%. Five curves for F1, F3, F3+, F7, and F7+ run from upper left to lower right. At the lower right, brown circles mark F7+ with repeated augmentation; pink, gray, and yellow stars mark ViT-g/14, ViT-G/14, and SoViT-400m/14. The end of the F7+ curve is at the same height as those stars.](/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png)](/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png)\n\n*Figure 1 from Smith et al., taken from [**PDF page 2**](https://arxiv.org/pdf/2310.16764v1#page=2). The original caption is omitted.*\n\nThe vertical axis is error, so lower is better. More pretraining compute leads to lower error after fine-tuning, across all the models.\n\nThe points at the lower right matter most. The largest F7+ is pretrained for 8 epochs, using about 110k TPU-v4 core hours for pretraining and 1.6k for fine-tuning, and reaches 90.3% ImageNet top-1. Adding repeated augmentation during fine-tuning raises it to 90.4%. For comparison, NFNet's previous best without extra data was 86.8%.\n\nNow look at ViT. The authors measure training speed on TPU-v4 using the original ViT authors' code, then convert the TPU-v3 compute reported in the ViT papers:\n\n| Model | ImageNet top-1 | Reported pretraining compute | Converted to TPU-v4 |\n| --- | ---: | --- | ---: |\n| ViT-g/14 | 90.2% | 210k TPU-v3 core hours | About 120k |\n| ViT-G/14 | 90.45% | Over 500k TPU-v3 core hours | About 280k |\n| SoViT-400m/14 | 90.3% | 230k TPU-v3 core hours | About 130k |\n| NFNet-F7+ | 90.3% | — | About 110k |\n| NFNet-F7+ (with RA) | 90.4% | — | About 110k |\n\n*Table: compiled from the main text on page 3 of the paper. ViT pretraining uses JFT-3B; NFNet uses JFT-4B.*\n\nAt similar compute, accuracy is similar too. ViT-G/14 gains 0.05 percentage points while using two and a half times F7+'s compute.\n\n### Lowest Loss Does Not Always Give the Best Fine-Tuning\n\nThere is another observation. At fixed pretraining compute, the model with the lowest JFT-4B validation loss need not be the best on ImageNet after fine-tuning. Fine-tuning tends to prefer a slightly larger model trained for slightly fewer epochs; a slightly higher pretraining learning rate sometimes helps too. The authors suggest that larger models have more capacity to adapt to the new task.\n\nThis is a useful caution when selecting models by scaling laws. The law predicts pretraining loss; the downstream metric you ultimately care about has a slightly different optimum.\n\n### Scope of the Comparison\n\nA few conditions matter here.\n\nThe ViT accuracies and compute come from the original papers, without retraining under one shared pipeline. Compute is estimated from the authors' speed measurements. NFNet is itself optimized for TPU-v4, and the authors acknowledge that it performs less well on other hardware. The only comparison task is ImageNet classification fine-tuning, without detection, segmentation, or multimodal tasks. JFT-4B is private, and the paper supplies neither code nor weights, so outsiders cannot reproduce the experiment.\n\nThe authors' own conclusion is restrained. Under a fair comparison, they find no strong evidence that pretrained ViTs outperform pretrained convolutional networks. They also acknowledge practical advantages of ViTs in some settings, such as reusing the same model components across modalities.\n\n## Conclusion\n\nThe discussion opens by saying that this paper reinforces Rich Sutton's “bitter lesson.” Given a reasonably designed model—enough expressive capacity and stable gradient propagation—performance is mainly determined by training compute and data.\n\nNFNet makes no architectural changes. Given compute similar to ViT's, it catches up. After years of architectural debate, this paper's answer is to align the budgets first.\n\nFor most people, though, the lesson is a little more bitter. JFT-4B is available only inside Google, and 110k TPU-v4 core hours is beyond the budget most labs can access.\n","imageSizes":{"/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png":{"width":942,"height":636},"/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png":{"width":954,"height":636},"/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png":{"width":1830,"height":828}}}