{"notes":[{"id":"classic-cnns/convnets-match-vits","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ConvNets Match ViTs","subtitle":"苦澀的教訓","author":"Z. Yuan","paperDate":"2023-10-25","publishedOn":"2025-11-17","paperTitle":"ConvNets Match Vision Transformers at Scale","paperUrl":"https://arxiv.org/abs/2310.16764","bibliography":{"title":"ConvNets Match Vision Transformers at Scale","authors":["Samuel L. Smith","Andrew Brock","Leonard Berrada","Soham De"],"url":"https://arxiv.org/abs/2310.16764","pdfUrl":"https://arxiv.org/pdf/2310.16764v1","codeUrl":null,"preprint":{"id":"2310.16764","submittedOn":"2023-10-25","url":"https://arxiv.org/abs/2310.16764"},"publication":null,"citation":"@misc{classiccnnsconvnetsmatchvits2023,\n  title = {{ConvNets Match Vision Transformers at Scale}},\n  author = {Samuel L. Smith and Andrew Brock and Leonard Berrada and Soham De},\n  year = {2023},\n  eprint = {2310.16764},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2310.16764}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnets-match-vits.0657c1e27561.bib"},"thumbnail":{"src":"/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png","width":1830,"height":828},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/convnets-match-vits.1166097df4cb.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"預訓練損失的縮放律","text":"預訓練損失的縮放律","depth":3},{"id":"學習率","text":"學習率","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-微調","text":"ImageNet 微調","depth":3},{"id":"損失最低的不一定微調最好","text":"損失最低的不一定微調最好","depth":3},{"id":"比較的範圍","text":"比較的範圍","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/vit","relation":"referenced","topic":""},{"id":"classic-cnns/nfnet","relation":"referenced","topic":""},{"id":"transformers/scaling_laws","relation":"referenced","topic":""}]},{"id":"vision-transformers/vit","category":"vision-transformers","legacyPath":"vision-transformers/2010-vit","legacySlug":"2010-vit","legacyPaths":[],"title":"ViT","subtitle":"新世界拓荒者","author":"Z. Yuan","paperDate":"2020-10-22","publishedOn":"2024-07-01","paperTitle":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","paperUrl":"https://arxiv.org/abs/2010.11929","bibliography":{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","authors":["Alexey Dosovitskiy","Lucas Beyer","Alexander Kolesnikov","Dirk Weissenborn","Xiaohua Zhai","Thomas Unterthiner","Mostafa Dehghani","Matthias Minderer","Georg Heigold","Sylvain Gelly","Jakob Uszkoreit","Neil Houlsby"],"url":"https://arxiv.org/abs/2010.11929","pdfUrl":"https://arxiv.org/pdf/2010.11929","codeUrl":null,"preprint":{"id":"2010.11929","submittedOn":"2020-10-22","url":"https://arxiv.org/abs/2010.11929"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://openreview.net/pdf?id=YicbFdNTTy"},"citation":"@misc{visiontransformersvit2020,\n  title = {{An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale}},\n  author = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},\n  year = {2020},\n  eprint = {2010.11929},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.11929}\n}\n","citationPath":"/content/papers/citations/vision-transformers/vit.8a478b78ec38.bib"},"thumbnail":{"src":"/content/papers/assets/82fb5d2f7ce623daf8d9ce146f096d22f2fb17d2532c5c4d613206f0424904aa.jpg","width":1472,"height":772},"locale":"zh-hant","path":"/content/papers/zh-hant/vision-transformers/vit.baf78e63c912.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"patchify","text":"Patchify","depth":3},{"id":"然後呢","text":"然後呢？","depth":3},{"id":"較少圖像歸納偏差","text":"較少圖像歸納偏差","depth":3},{"id":"預訓練資料規模的影響","text":"預訓練資料規模的影響","depth":3},{"id":"還能再更大","text":"還能再更大","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"vit-看到了什麼","text":"ViT 看到了什麼？","depth":3},{"id":"用-cls-預測還是用-gap-預測","text":"用 [CLS] 預測還是用 GAP 預測？","depth":3},{"id":"自監督訓練","text":"自監督訓練","depth":3},{"id":"其他注意事項","text":"其他注意事項","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/deit","relation":"next","topic":"Transformer 如何學習視覺表徵"},{"id":"vision-transformers/pvt","relation":"comparison","topic":"多尺度視覺 Transformer"},{"id":"vision-transformers/swin-transformer","relation":"comparison","topic":"多尺度視覺 Transformer"}]},{"id":"classic-cnns/nfnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NFNet","subtitle":"沒有 BN 的日子","author":"Z. Yuan","paperDate":"2021-02-11","publishedOn":"2025-01-21","world":"external/nfnet","paperTitle":"High-Performance Large-Scale Image Recognition Without Normalization","paperUrl":"https://arxiv.org/abs/2102.06171","bibliography":{"title":"High-Performance Large-Scale Image Recognition Without Normalization","authors":["Andrew Brock","Soham De","Samuel L. Smith","Karen Simonyan"],"url":"https://arxiv.org/abs/2102.06171","pdfUrl":"https://arxiv.org/pdf/2102.06171v1","codeUrl":null,"preprint":{"id":"2102.06171","submittedOn":"2021-02-11","url":"https://arxiv.org/abs/2102.06171"},"publication":{"venue":"ICML","year":2021,"kind":"conference","url":"https://proceedings.mlr.press/v139/brock21a.html"},"citation":"@misc{classiccnnsnfnet2021,\n  title = {{High-Performance Large-Scale Image Recognition Without Normalization}},\n  author = {Andrew Brock and Soham De and Samuel L. Smith and Karen Simonyan},\n  year = {2021},\n  eprint = {2102.06171},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.06171}\n}\n","citationPath":"/content/papers/citations/classic-cnns/nfnet.91baba59df96.bib"},"thumbnail":{"src":"/content/papers/assets/7552bff30433f4f33bec5edb2421acd94d61d452888dde6df17557778e8fb1c6.png","width":1004,"height":783},"locale":"zh-hant","path":"/content/papers/zh-hant/classic-cnns/nfnet.2b49d5235475.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"batchnorm-到底幫了什麼","text":"BatchNorm 到底幫了什麼","depth":3},{"id":"平均值偏移長什麼樣子","text":"平均值偏移長什麼樣子","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"先把-nf-resnet-補齊","text":"先把 NF-ResNet 補齊","depth":3},{"id":"自適應梯度裁剪","text":"自適應梯度裁剪","depth":3},{"id":"nfnet-的架構","text":"NFNet 的架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"每一步改動的貢獻","text":"每一步改動的貢獻","depth":3},{"id":"拿掉-bn-真的比較快嗎","text":"拿掉 BN 真的比較快嗎","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"預訓練之後再微調","text":"預訓練之後再微調","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/batchnorm","relation":"referenced","topic":""}]},{"id":"transformers/scaling_laws","category":"transformers","legacyPath":"transformers/2001-scaling_laws","legacySlug":"2001-scaling_laws","legacyPaths":[],"title":"Scaling Laws","subtitle":"模型的縮放律","author":"Z. Yuan","paperDate":"2020-01-23","publishedOn":"2024-05-24","paperTitle":"Scaling Laws for Neural Language Models","paperUrl":"https://arxiv.org/abs/2001.08361","bibliography":{"title":"Scaling Laws for Neural Language Models","authors":["Jared Kaplan","Sam McCandlish","Tom Henighan","Tom B. Brown","Benjamin Chess","Rewon Child","Scott Gray","Alec Radford","Jeffrey Wu","Dario Amodei"],"url":"https://arxiv.org/abs/2001.08361","pdfUrl":"https://arxiv.org/pdf/2001.08361","codeUrl":null,"preprint":{"id":"2001.08361","submittedOn":"2020-01-23","url":"https://arxiv.org/abs/2001.08361"},"publication":null,"citation":"@misc{transformersscalinglaws2020,\n  title = {{Scaling Laws for Neural Language Models}},\n  author = {Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei},\n  year = {2020},\n  eprint = {2001.08361},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2001.08361}\n}\n","citationPath":"/content/papers/citations/transformers/scaling_laws.a1ab0069ac66.bib"},"thumbnail":{"src":"/content/papers/assets/1a680a937f0c09dd160f2661fc04c4598f9bec153e122a7b92c264ef213558c1.jpg","width":1696,"height":544},"locale":"zh-hant","path":"/content/papers/zh-hant/transformers/scaling_laws.0fd528ed368f.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"訓練過程","text":"訓練過程","depth":3},{"id":"數據集","text":"數據集","depth":3},{"id":"變化因素","text":"變化因素","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"規模大於一切","text":"規模大於一切","depth":3},{"id":"冪定律","text":"冪定律","depth":3},{"id":"過擬合的普遍性","text":"過擬合的普遍性","depth":3},{"id":"測試表現與下游表現","text":"測試表現與下游表現","depth":3},{"id":"樣本效率","text":"樣本效率","depth":3},{"id":"收斂是一件低效的事情","text":"收斂是一件低效的事情","depth":3},{"id":"最佳批量大小","text":"最佳批量大小","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"transformers/gpt_3","relation":"background","topic":"模型大小與訓練資料量的取捨"},{"id":"transformers/chinchilla","relation":"next","topic":"模型大小與訓練資料量的取捨"}]}],"text":"「卷積網路在中小型資料集上表現不錯，可是資料一到網路規模，就比不上 Vision Transformer。」\n\n論文的第一句就是這個說法，作者說很多研究者都這麼相信。ViT 的參數一路做到十億以上、在幾十億張圖上預訓練，卷積網路這邊卻很少有人用同樣的規模訓練過。\n\n\n> [!TIP]\n> \n> ViT 怎麼把圖片切成 patch 送進 Transformer，可以看這篇：\n> \n> - **[[20.10] ViT：新世界拓荒者](/blog/papers/vision-transformers/vit)**\n\n\n這篇來自 Google DeepMind，作者是 Samuel L. Smith、Andrew Brock、Leonard Berrada 和 Soham De。Andrew Brock 是 NFNet 的第一作者。正文只有四頁，作者在致謝裡自己稱它為一篇 note。它只問一個問題，給卷積網路和 ViT 差不多的預訓練算力，ViT 還會比較好嗎？\n\n## 定義問題\n\n作者認為，支持這個看法的證據其實很少。\n\n第一，很多 ViT 論文拿來比較的卷積網路，是最原始的 ResNet，等於拿新配方和舊基準比。\n\n第二，算力差太多。最強的 ViT 預訓練用了超過 50 萬 TPU-v3 core hours，遠超過卷積網路曾經用過的預訓練算力。\n\n所以作者不改架構，也不改訓練流程，只調學習率和 epoch 數這類簡單的超參數，把 NFNet 拿到同樣的規模上訓練，看看會發生什麼事。NFNet 是 2021 年的純卷積網路，和第一批 ViT 論文差不多同時發表，作者形容它是最後一個在 ImageNet 上刷新最佳成績的卷積網路。\n\n\n> [!TIP]\n> \n> NFNet 拿掉了 BatchNorm，靠自適應梯度裁剪撐住大批次訓練，細節可以看這篇：\n> \n> - **[[21.02] NFNet：沒有 BN 的日子](/blog/papers/classic-cnns/nfnet)**\n\n\n## 解決問題\n\n### 實驗設定\n\n預訓練資料是 JFT-4B，大約 40 億張有標籤的圖片，分成 3 萬個類別，訓練前先移除了和 ImageNet 訓練集、驗證集近似重複的圖片。這份資料集不公開。\n\n模型是 NFNet 家族裡不同深度和寬度的版本。F1 的深度是 F0 的兩倍，F3 是 F1 的兩倍，F7 又是 F3 的兩倍；F3+ 和 F7+ 深度不變，寬度加大。\n\n每個模型都在 0.25 到 8 個 epoch 之間訓練好幾次，學習率用 cosine 衰減。優化器是帶動量的 SGD 加上 NFNet 原本的自適應梯度裁剪（AGC），批次 4096，訓練解析度 224×224。\n\n算力用 TPU-v4 core hours 計算，範圍從 0.4k 到 110k。TPU-v4 單核的理論算力大約是 TPU-v3 的兩倍，記憶體差不多，所以拿 ViT 論文的 TPU-v3 數字來比時要換算，後面會再提到。\n\n### 預訓練損失的縮放律\n\n[![原論文 Figure 2：雙對數座標，橫軸是預訓練用的 TPU-v4 core hours（約 10³ 到 10⁵），縱軸是 JFT-4B 保留集上的驗證損失（約 2.4 到 3.0）。F0、F1、F3、F3+、F7、F7+ 六種模型各一條折線，每條線上的點是不同的 epoch 數；所有曲線大致沿著同一條從左上到右下的直線排列，F7+ 的線在中段有些不平整。](/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png)](/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png)\n\n*原論文 Figure 2（Smith 等人），取自 [**PDF 第 3 頁**](https://arxiv.org/pdf/2310.16764v1#page=3)。圖說未收錄。*\n\n圖中每條折線是一個模型，線上的點是不同的 epoch 數。兩個軸都是對數刻度，所有曲線的下緣大致排成一條直線，也就是驗證損失和算力之間存在冪次關係。語言模型的縮放律長的就是這個樣子，NFNet 也一樣。\n\n每個算力預算都有一個損失最低的模型和 epoch 數。作者發現兩者會一起變大，經驗法則是模型大小和訓練 epoch 數用同樣的速度放大，這也和語言模型的 Chinchilla 結論一致。算力超過大約 5k TPU-v4 core hours 之後，最佳的 epoch 數就會超過 1。\n\nF7+ 那條線不太平整。作者解釋，資料讀取流程在訓練中斷重啟時，不保證每張圖在一個 epoch 裡剛好被看到一次，重啟幾次之後，有些樣本就被看得比較少。\n\n\n> [!TIP]\n> \n> 語言模型的縮放律和 Chinchilla，可以參考我們的筆記：\n> \n> - **[[20.01] Scaling Laws：模型的縮放律](/blog/papers/transformers/scaling_laws)**\n> - **[[22.03] Chinchilla：栗鼠之眼](/blog/papers/transformers/chinchilla)**\n\n\n### 學習率\n\n[![原論文 Figure 3：橫軸是訓練 epoch 數（0.25 到 8，對數刻度），縱軸是最佳學習率（0.4 到 1.6）。F0、F3、F7+ 三條線在 0.25 到 1 個 epoch 時都停在 1.6；F7+ 的虛線在 2 個 epoch 掉到 0.8、4 個 epoch 掉到 0.4；F3 在 2 到 4 個 epoch 是 0.8、8 個 epoch 是 0.4；F0 一直維持 1.6 到 4 個 epoch，8 個 epoch 才降到 0.8。](/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png)](/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png)\n\n*原論文 Figure 3（Smith 等人），取自 [**PDF 第 3 頁**](https://arxiv.org/pdf/2310.16764v1#page=3)。圖說未收錄。*\n\n學習率在一個以 2 倍為間隔的對數網格上搜尋。訓練 1 個 epoch 以內，三個模型的最佳學習率都是 1.6；epoch 數變多，最佳學習率開始下降，模型越大降得越早。\n\n這張圖的實用價值在後半句。作者說，既然最佳學習率隨著模型和 epoch 數緩慢、單調地下降，實務上大約試兩次就能找到它。\n\n## 討論\n\n### ImageNet 微調\n\n預訓練完的模型拿到 ImageNet 上微調 50 個 epoch，用 SAM 優化器加上 stochastic depth 和 dropout，訓練解析度 384×384，評估解析度 480×480。\n\n[![原論文 Figure 1：雙對數座標，橫軸是預訓練的 TPU-v4 core hours（約 10³ 到 10⁵），縱軸是 ImageNet top-1 錯誤率（約 9.5% 到 14%）。F1、F3、F3+、F7、F7+ 五條折線從左上往右下延伸；右下角另有幾個點，咖啡色圓點是加了 repeated augmentation 的 F7+，粉紅、灰色、黃色星號分別是 ViT-g/14、ViT-G/14 和 SoViT-400m/14，F7+ 的末端和這幾顆星號落在同一個高度。](/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png)](/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png)\n\n*原論文 Figure 1（Smith 等人），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/2310.16764v1#page=2)。圖說未收錄。*\n\n縱軸是錯誤率，越低越好。預訓練的算力越多，微調後的錯誤率就越低，這個趨勢在所有模型上都看得到。\n\n最右下角那幾個點是重點。最大的 F7+ 預訓練 8 個 epoch，用了大約 110k TPU-v4 core hours 預訓練、1.6k 微調，ImageNet top-1 是 90.3%。微調時再加上 repeated augmentation，可以到 90.4%。對照一下，不用額外資料的 NFNet，之前最好的紀錄是 86.8%。\n\n再看 ViT。作者用 ViT 原作者的程式碼在 TPU-v4 上量了訓練速度，把 ViT 論文報告的 TPU-v3 算力換算過來：\n\n| 模型 | ImageNet top-1 | 論文報告的預訓練算力 | 換算成 TPU-v4 |\n| --- | ---: | --- | ---: |\n| ViT-g/14 | 90.2% | 210k TPU-v3 core hours | 約 120k |\n| ViT-G/14 | 90.45% | 超過 500k TPU-v3 core hours | 約 280k |\n| SoViT-400m/14 | 90.3% | 230k TPU-v3 core hours | 約 130k |\n| NFNet-F7+ | 90.3% | — | 約 110k |\n| NFNet-F7+（加 RA） | 90.4% | — | 約 110k |\n\n*表：整理自原論文第 3 頁的正文。ViT 的預訓練資料是 JFT-3B，NFNet 是 JFT-4B。*\n\n算力差不多的情況下，準確率也差不多。ViT-G/14 多了 0.05 個百分點，用的算力是 F7+ 的兩倍半。\n\n### 損失最低的不一定微調最好\n\n還有一個觀察。固定預訓練算力，JFT-4B 上驗證損失最低的模型，微調之後不一定是 ImageNet 最好的。微調比較偏好「稍微大一點的模型、稍微少一點的 epoch」，有時候預訓練學習率稍大一些也比較好。作者的解釋是，大模型容量比較多，比較能適應新任務。\n\n這對想照著縮放律挑模型的人是個提醒。縮放律預測的是預訓練損失，你最後在乎的下游表現，最佳點會偏一點。\n\n### 比較的範圍\n\n這組比較有幾個條件要記著。\n\nViT 那邊的準確率和算力來自原論文，沒有在同一套流程下重新訓練；算力是作者量測後估出來的，而 NFNet 本來就是為 TPU-v4 優化的架構，作者也承認它在其他硬體上表現比較差。比較的任務只有 ImageNet 分類微調，沒有偵測、分割或多模態。JFT-4B 不公開，論文也沒有附上程式碼或權重，外人無法重現。\n\n作者自己的結論也很克制。他們說的是，公平比較的話，沒有強力證據顯示預訓練的 ViT 比預訓練的卷積網路好；同時也承認 ViT 在某些情境有實際的優勢，例如不同模態可以共用相同的模型元件。\n\n## 結論\n\n作者在討論的第一句就寫了，這篇強化了 Rich Sutton 說的「苦澀的教訓」。只要模型設計得合理，也就是表達能力夠、梯度能穩定傳遞，決定表現的主要是訓練用的算力和資料。\n\nNFNet 沒有改任何架構，只是拿到和 ViT 差不多的算力，就追上了 ViT。架構之爭吵了好幾年，這篇的答案是先把預算對齊再說。\n\n只是這個教訓對大多數人來說更苦一點。JFT-4B 只在 Google 內部，110k 個 TPU-v4 core hours 也不是一般實驗室排得到的預算。\n","imageSizes":{"/content/papers/assets/2fa2168f7bc3a058a806eeda379d056c232c85461c0928339b15af2ecdd2da7b.png":{"width":942,"height":636},"/content/papers/assets/554306115d516021efec1a96e4e6c57bb4da8ff8846a0b351408df22429c19a4.png":{"width":954,"height":636},"/content/papers/assets/3e69462cce7a0dbd3fc43590ac1f01d23631b4677d04be8bdeae74e6b987e620.png":{"width":1830,"height":828}}}