{"notes":[{"id":"classic-cnns/convnets-match-vits","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ConvNets Match ViTs","subtitle":"苦涩的教训","author":"Z. Yuan","paperDate":"2023-10-25","publishedOn":"2025-11-17","paperTitle":"ConvNets Match Vision Transformers at Scale","paperUrl":"https://arxiv.org/abs/2310.16764","bibliography":{"title":"ConvNets Match Vision Transformers at Scale","authors":["Samuel L. Smith","Andrew Brock","Leonard Berrada","Soham De"],"url":"https://arxiv.org/abs/2310.16764","pdfUrl":"https://arxiv.org/pdf/2310.16764v1","codeUrl":null,"preprint":{"id":"2310.16764","submittedOn":"2023-10-25","url":"https://arxiv.org/abs/2310.16764"},"publication":null,"citation":"@misc{classiccnnsconvnetsmatchvits2023,\n  title = {{ConvNets Match Vision Transformers at Scale}},\n  author = {Samuel L. Smith and Andrew Brock and Leonard Berrada and Soham De},\n  year = {2023},\n  eprint = {2310.16764},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2310.16764}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnets-match-vits.0657c1e27561.bib"},"thumbnail":{"src":"/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png","width":1830,"height":828},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/convnets-match-vits.a3dd02ca4900.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"预训练损失的缩放律","text":"预训练损失的缩放律","depth":3},{"id":"学习率","text":"学习率","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"imagenet-微调","text":"ImageNet 微调","depth":3},{"id":"损失最低的不一定微调最好","text":"损失最低的不一定微调最好","depth":3},{"id":"比较的范围","text":"比较的范围","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/vit","relation":"referenced","topic":""},{"id":"classic-cnns/nfnet","relation":"referenced","topic":""},{"id":"transformers/scaling_laws","relation":"referenced","topic":""}]},{"id":"vision-transformers/vit","category":"vision-transformers","legacyPath":"vision-transformers/2010-vit","legacySlug":"2010-vit","legacyPaths":[],"title":"ViT","subtitle":"新世界拓荒者","author":"Z. Yuan","paperDate":"2020-10-22","publishedOn":"2024-07-01","paperTitle":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","paperUrl":"https://arxiv.org/abs/2010.11929","bibliography":{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","authors":["Alexey Dosovitskiy","Lucas Beyer","Alexander Kolesnikov","Dirk Weissenborn","Xiaohua Zhai","Thomas Unterthiner","Mostafa Dehghani","Matthias Minderer","Georg Heigold","Sylvain Gelly","Jakob Uszkoreit","Neil Houlsby"],"url":"https://arxiv.org/abs/2010.11929","pdfUrl":"https://arxiv.org/pdf/2010.11929","codeUrl":null,"preprint":{"id":"2010.11929","submittedOn":"2020-10-22","url":"https://arxiv.org/abs/2010.11929"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://openreview.net/pdf?id=YicbFdNTTy"},"citation":"@misc{visiontransformersvit2020,\n  title = {{An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale}},\n  author = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},\n  year = {2020},\n  eprint = {2010.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.11929}\n}\n","citationPath":"/content/papers/citations/vision-transformers/vit.8a478b78ec38.bib"},"thumbnail":{"src":"/content/papers/assets/82fb5d2f7ce623daf8d9ce146f096d22f2fb17d2532c5c4d613206f0424904aa.jpg","width":1472,"height":772},"locale":"zh-hans","path":"/content/papers/zh-hans/vision-transformers/vit.254b89f47360.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型架构","text":"模型架构","depth":3},{"id":"patchify","text":"Patchify","depth":3},{"id":"然后呢","text":"然后呢？","depth":3},{"id":"较少图像归纳偏差","text":"较少图像归纳偏差","depth":3},{"id":"预训练数据规模的影响","text":"预训练数据规模的影响","depth":3},{"id":"还能再更大","text":"还能再更大","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"vit-看到了什么","text":"ViT 看到了什么？","depth":3},{"id":"用-cls-预测还是用-gap-预测","text":"用 [CLS] 预测还是用 GAP 预测？","depth":3},{"id":"自监督训练","text":"自监督训练","depth":3},{"id":"其他注意事项","text":"其他注意事项","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/deit","relation":"next","topic":"Transformer 如何学习视觉表示"},{"id":"vision-transformers/pvt","relation":"comparison","topic":"多尺度视觉 Transformer"},{"id":"vision-transformers/swin-transformer","relation":"comparison","topic":"多尺度视觉 Transformer"}]},{"id":"classic-cnns/nfnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NFNet","subtitle":"没有 BN 的日子","author":"Z. Yuan","paperDate":"2021-02-11","publishedOn":"2025-01-21","world":"external/nfnet","paperTitle":"High-Performance Large-Scale Image Recognition Without Normalization","paperUrl":"https://arxiv.org/abs/2102.06171","bibliography":{"title":"High-Performance Large-Scale Image Recognition Without Normalization","authors":["Andrew Brock","Soham De","Samuel L. Smith","Karen Simonyan"],"url":"https://arxiv.org/abs/2102.06171","pdfUrl":"https://arxiv.org/pdf/2102.06171v1","codeUrl":null,"preprint":{"id":"2102.06171","submittedOn":"2021-02-11","url":"https://arxiv.org/abs/2102.06171"},"publication":{"venue":"ICML","year":2021,"kind":"conference","url":"https://proceedings.mlr.press/v139/brock21a.html"},"citation":"@misc{classiccnnsnfnet2021,\n  title = {{High-Performance Large-Scale Image Recognition Without Normalization}},\n  author = {Andrew Brock and Soham De and Samuel L. Smith and Karen Simonyan},\n  year = {2021},\n  eprint = {2102.06171},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.06171}\n}\n","citationPath":"/content/papers/citations/classic-cnns/nfnet.91baba59df96.bib"},"thumbnail":{"src":"/content/papers/assets/7552bff30433f4f33bec5edb2421acd94d61d452888dde6df17557778e8fb1c6.png","width":1004,"height":783},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/nfnet.38a489ed8238.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"batchnorm-到底帮了什么","text":"BatchNorm 到底帮了什么","depth":3},{"id":"均值偏移长什么样子","text":"均值偏移长什么样子","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"先把-nf-resnet-补齐","text":"先把 NF-ResNet 补齐","depth":3},{"id":"自适应梯度裁剪","text":"自适应梯度裁剪","depth":3},{"id":"nfnet-的架构","text":"NFNet 的架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"每一步改动的贡献","text":"每一步改动的贡献","depth":3},{"id":"拿掉-bn-真的比较快吗","text":"拿掉 BN 真的比较快吗","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"预训练之后再微调","text":"预训练之后再微调","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/batchnorm","relation":"referenced","topic":""}]},{"id":"transformers/scaling_laws","category":"transformers","legacyPath":"transformers/2001-scaling_laws","legacySlug":"2001-scaling_laws","legacyPaths":[],"title":"Scaling Laws","subtitle":"模型的缩放律","author":"Z. Yuan","paperDate":"2020-01-23","publishedOn":"2024-05-24","paperTitle":"Scaling Laws for Neural Language Models","paperUrl":"https://arxiv.org/abs/2001.08361","bibliography":{"title":"Scaling Laws for Neural Language Models","authors":["Jared Kaplan","Sam McCandlish","Tom Henighan","Tom B. Brown","Benjamin Chess","Rewon Child","Scott Gray","Alec Radford","Jeffrey Wu","Dario Amodei"],"url":"https://arxiv.org/abs/2001.08361","pdfUrl":"https://arxiv.org/pdf/2001.08361","codeUrl":null,"preprint":{"id":"2001.08361","submittedOn":"2020-01-23","url":"https://arxiv.org/abs/2001.08361"},"publication":null,"citation":"@misc{transformersscalinglaws2020,\n  title = {{Scaling Laws for Neural Language Models}},\n  author = {Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei},\n  year = {2020},\n  eprint = {2001.08361},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2001.08361}\n}\n","citationPath":"/content/papers/citations/transformers/scaling_laws.a1ab0069ac66.bib"},"thumbnail":{"src":"/content/papers/assets/1a680a937f0c09dd160f2661fc04c4598f9bec153e122a7b92c264ef213558c1.jpg","width":1696,"height":544},"locale":"zh-hans","path":"/content/papers/zh-hans/transformers/scaling_laws.ac367d68c252.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"训练过程","text":"训练过程","depth":3},{"id":"数据集","text":"数据集","depth":3},{"id":"变化因素","text":"变化因素","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"规模大于一切","text":"规模大于一切","depth":3},{"id":"幂定律","text":"幂定律","depth":3},{"id":"过拟合的普遍性","text":"过拟合的普遍性","depth":3},{"id":"测试表现与下游表现","text":"测试表现与下游表现","depth":3},{"id":"样本效率","text":"样本效率","depth":3},{"id":"收敛是一件低效的事情","text":"收敛是一件低效的事情","depth":3},{"id":"最佳批量大小","text":"最佳批量大小","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/gpt_3","relation":"background","topic":"模型大小与训练数据量的取舍"},{"id":"transformers/chinchilla","relation":"next","topic":"模型大小与训练数据量的取舍"}]}],"text":"「卷积网络在中小型数据集上表现不错，可是数据一到网络规模，就比不上 Vision Transformer。」\n\n论文的第一句就是这个说法，作者说很多研究者都这么相信。ViT 的参数一路做到十亿以上、在几十亿张图上预训练，卷积网络这边却很少有人用同样的规模训练过。\n\n\n> [!TIP]\n> \n> ViT 怎么把图片切成 patch 送进 Transformer，可以看这篇：\n> \n> - **[[20.10] ViT：新世界拓荒者](/blog/papers/vision-transformers/vit)**\n\n\n这篇来自 Google DeepMind，作者是 Samuel L. Smith、Andrew Brock、Leonard Berrada 和 Soham De。Andrew Brock 是 NFNet 的第一作者。正文只有四页，作者在致谢里自己称它为一篇 note。它只问一个问题，给卷积网络和 ViT 差不多的预训练算力，ViT 还会比较好吗？\n\n## 定义问题\n\n作者认为，支持这个看法的证据其实很少。\n\n第一，很多 ViT 论文拿来比较的卷积网络，是最原始的 ResNet，等于拿新配方和旧基准比。\n\n第二，算力差太多。最强的 ViT 预训练用了超过 50 万 TPU-v3 core hours，远超过卷积网络曾经用过的预训练算力。\n\n所以作者不改架构，也不改训练流程，只调学习率和 epoch 数这类简单的超参数，把 NFNet 拿到同样的规模上训练，看看会发生什么事。NFNet 是 2021 年的纯卷积网络，和第一批 ViT 论文差不多同时发表，作者形容它是最后一个在 ImageNet 上刷新最佳成绩的卷积网络。\n\n\n> [!TIP]\n> \n> NFNet 拿掉了 BatchNorm，靠自适应梯度裁剪撑住大批量训练，细节可以看这篇：\n> \n> - **[[21.02] NFNet：没有 BN 的日子](/blog/papers/classic-cnns/nfnet)**\n\n\n## 解决问题\n\n### 实验设置\n\n预训练数据是 JFT-4B，大约 40 亿张有标签的图片，分成 3 万个类别，训练前先移除了和 ImageNet 训练集、验证集近似重复的图片。这份数据集不公开。\n\n模型是 NFNet 家族里不同深度和宽度的版本。F1 的深度是 F0 的两倍，F3 是 F1 的两倍，F7 又是 F3 的两倍；F3+ 和 F7+ 深度不变，宽度加大。\n\n每个模型都在 0.25 到 8 个 epoch 之间训练好几次，学习率用 cosine 衰减。优化器是带动量的 SGD 加上 NFNet 原本的自适应梯度裁剪（AGC），批量 4096，训练分辨率 224×224。\n\n算力用 TPU-v4 core hours 计算，范围从 0.4k 到 110k。TPU-v4 单核的理论算力大约是 TPU-v3 的两倍，内存差不多，所以拿 ViT 论文的 TPU-v3 数字来比时要换算，后面会再提到。\n\n### 预训练损失的缩放律\n\n[![原论文 Figure 2：双对数坐标，横轴是预训练用的 TPU-v4 core hours（约 10³ 到 10⁵），纵轴是 JFT-4B 保留集上的验证损失（约 2.4 到 3.0）。F0、F1、F3、F3+、F7、F7+ 六种模型各一条折线，每条线上的点是不同的 epoch 数；所有曲线大致沿着同一条从左上到右下的直线排列，F7+ 的线在中段有些不平整。](/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png)](/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png)\n\n*原论文 Figure 2（Smith 等人），取自 [**PDF 第 3 页**](https://arxiv.org/pdf/2310.16764v1#page=3)。原图注未收录。*\n\n图中每条折线是一个模型，线上的点是不同的 epoch 数。两个轴都是对数刻度，所有曲线的下缘大致排成一条直线，也就是验证损失和算力之间存在幂次关系。语言模型的缩放律长的就是这个样子，NFNet 也一样。\n\n每个算力预算都有一个损失最低的模型和 epoch 数。作者发现两者会一起变大，经验法则是模型大小和训练 epoch 数用同样的速度放大，这也和语言模型的 Chinchilla 结论一致。算力超过大约 5k TPU-v4 core hours 之后，最佳的 epoch 数就会超过 1。\n\nF7+ 那条线不太平整。作者解释，数据读取流程在训练中断重启时，不保证每张图在一个 epoch 里刚好被看到一次，重启几次之后，有些样本就被看得比较少。\n\n\n> [!TIP]\n> \n> 语言模型的缩放律和 Chinchilla，可以参考我们的笔记：\n> \n> - **[[20.01] Scaling Laws：模型的缩放律](/blog/papers/transformers/scaling_laws)**\n> - **[[22.03] Chinchilla：栗鼠之眼](/blog/papers/transformers/chinchilla)**\n\n\n### 学习率\n\n[![原论文 Figure 3：横轴是训练 epoch 数（0.25 到 8，对数刻度），纵轴是最佳学习率（0.4 到 1.6）。F0、F3、F7+ 三条线在 0.25 到 1 个 epoch 时都停在 1.6；F7+ 的虚线在 2 个 epoch 掉到 0.8、4 个 epoch 掉到 0.4；F3 在 2 到 4 个 epoch 是 0.8、8 个 epoch 是 0.4；F0 一直维持 1.6 到 4 个 epoch，8 个 epoch 才降到 0.8。](/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png)](/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png)\n\n*原论文 Figure 3（Smith 等人），取自 [**PDF 第 3 页**](https://arxiv.org/pdf/2310.16764v1#page=3)。原图注未收录。*\n\n学习率在一个以 2 倍为间隔的对数网格上搜索。训练 1 个 epoch 以内，三个模型的最佳学习率都是 1.6；epoch 数变多，最佳学习率开始下降，模型越大降得越早。\n\n这张图的实用价值在后半句。作者说，既然最佳学习率随着模型和 epoch 数缓慢、单调地下降，实际使用时大约试两次就能找到它。\n\n## 讨论\n\n### ImageNet 微调\n\n预训练完的模型拿到 ImageNet 上微调 50 个 epoch，用 SAM 优化器加上 stochastic depth 和 dropout，训练分辨率 384×384，评估分辨率 480×480。\n\n[![原论文 Figure 1：双对数坐标，横轴是预训练的 TPU-v4 core hours（约 10³ 到 10⁵），纵轴是 ImageNet top-1 错误率（约 9.5% 到 14%）。F1、F3、F3+、F7、F7+ 五条折线从左上往右下延伸；右下角另有几个点，咖啡色圆点是加了 repeated augmentation 的 F7+，粉红、灰色、黄色星号分别是 ViT-g/14、ViT-G/14 和 SoViT-400m/14，F7+ 的末端和这几颗星号落在同一个高度。](/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png)](/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png)\n\n*原论文 Figure 1（Smith 等人），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/2310.16764v1#page=2)。原图注未收录。*\n\n纵轴是错误率，越低越好。预训练的算力越多，微调后的错误率就越低，这个趋势在所有模型上都看得到。\n\n最右下角那几个点是重点。最大的 F7+ 预训练 8 个 epoch，用了大约 110k TPU-v4 core hours 预训练、1.6k 微调，ImageNet top-1 是 90.3%。微调时再加上 repeated augmentation，可以到 90.4%。对照一下，不用额外数据的 NFNet，之前最好的记录是 86.8%。\n\n再看 ViT。作者用 ViT 原作者的代码在 TPU-v4 上量了训练速度，把 ViT 论文报告的 TPU-v3 算力换算过来：\n\n| 模型 | ImageNet top-1 | 论文报告的预训练算力 | 换算成 TPU-v4 |\n| --- | ---: | --- | ---: |\n| ViT-g/14 | 90.2% | 210k TPU-v3 core hours | 约 120k |\n| ViT-G/14 | 90.45% | 超过 500k TPU-v3 core hours | 约 280k |\n| SoViT-400m/14 | 90.3% | 230k TPU-v3 core hours | 约 130k |\n| NFNet-F7+ | 90.3% | — | 约 110k |\n| NFNet-F7+（加 RA） | 90.4% | — | 约 110k |\n\n*表：整理自原论文第 3 页的正文。ViT 的预训练数据是 JFT-3B，NFNet 是 JFT-4B。*\n\n算力差不多的情况下，准确率也差不多。ViT-G/14 多了 0.05 个百分点，用的算力是 F7+ 的两倍半。\n\n### 损失最低的不一定微调最好\n\n还有一个观察。固定预训练算力，JFT-4B 上验证损失最低的模型，微调之后不一定是 ImageNet 最好的。微调比较偏好「稍微大一点的模型、稍微少一点的 epoch」，有时候预训练学习率稍大一些也比较好。作者的解释是，大模型容量比较多，比较能适应新任务。\n\n这对想照着缩放律挑模型的人是个提醒。缩放律预测的是预训练损失，你最后在乎的下游表现，最佳点会偏一点。\n\n### 比较的范围\n\n这组比较有几个条件要记着。\n\nViT 那边的准确率和算力来自原论文，没有在同一套流程下重新训练；算力是作者测量后估出来的，而 NFNet 本来就是为 TPU-v4 优化的架构，作者也承认它在其他硬件上表现比较差。比较的任务只有 ImageNet 分类微调，没有检测、分割或多模态。JFT-4B 不公开，论文也没有附上代码或权重，外人无法复现。\n\n作者自己的结论也很克制。他们说的是，公平比较的话，没有强力证据显示预训练的 ViT 比预训练的卷积网络好；同时也承认 ViT 在某些情境有实际的优势，例如不同模态可以共用相同的模型组件。\n\n## 结论\n\n作者在讨论的第一句就写了，这篇强化了 Rich Sutton 说的「苦涩的教训」。只要模型设计得合理，也就是表达能力够、梯度能稳定传递，决定表现的主要是训练用的算力和数据。\n\nNFNet 没有改任何架构，只是拿到和 ViT 差不多的算力，就追上了 ViT。架构之争吵了好几年，这篇的答案是先把预算对齐再说。\n\n只是这个教训对大多数人来说更苦一点。JFT-4B 只在 Google 内部，110k 个 TPU-v4 core hours 也不是一般实验室排得到的预算。\n","imageSizes":{"/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png":{"width":942,"height":636},"/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png":{"width":954,"height":636},"/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png":{"width":1830,"height":828}}}