{"notes":[{"id":"vision-transformers/poolformer","category":"vision-transformers","legacyPath":"vision-transformers/2111-poolformer","legacySlug":"2111-poolformer","legacyPaths":[],"title":"PoolFormer","subtitle":"你需要的是框架！","author":"Z. Yuan","paperDate":"2021-11-22","publishedOn":"2024-07-10","paperTitle":"MetaFormer Is Actually What You Need for Vision","paperUrl":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","bibliography":{"title":"MetaFormer Is Actually What You Need for Vision","authors":["Weihao Yu","Mi Luo","Pan Zhou","Chenyang Si","Yichen Zhou","Xinchao Wang","Jiashi Feng","Shuicheng Yan"],"url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2022/papers/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.pdf","codeUrl":"https://github.com/sail-sg/poolformer","preprint":{"id":"2111.11418","submittedOn":"2021-11-22","url":"https://arxiv.org/abs/2111.11418"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html"},"citation":"@misc{visiontransformerspoolformer2021,\n  title = {{MetaFormer Is Actually What You Need for Vision}},\n  author = {Weihao Yu and Mi Luo and Pan Zhou and Chenyang Si and Yichen Zhou and Xinchao Wang and Jiashi Feng and Shuicheng Yan},\n  year = {2021},\n  eprint = {2111.11418},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.11418}\n}\n","citationPath":"/content/papers/citations/vision-transformers/poolformer.f2f96629d283.bib"},"thumbnail":{"src":"/content/papers/assets/3cb141e9bb1247410c8214276b21b28ed1d536060d470406f1bfb0c88fe68df2.jpg","width":1498,"height":548},"locale":"zh-hant","path":"/content/papers/zh-hant/vision-transformers/poolformer.cc4d5c111221.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"input-embedding","text":"Input Embedding","depth":3},{"id":"token-mixer","text":"Token-Mixer","depth":3},{"id":"channel-mlp","text":"Channel MLP","depth":3},{"id":"或許-token-mixer-不是重點","text":"或許 Token-Mixer 不是重點？","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"模型配置","text":"模型配置","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"imagenet-實驗","text":"ImageNet 實驗","depth":3},{"id":"消融實驗","text":"消融實驗","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/mlp-mixer","relation":"background","topic":"把骨架與 Token 混合方式分開看"},{"id":"vision-transformers/caformer","relation":"next","topic":"把骨架與 Token 混合方式分開看"}]},{"id":"vision-transformers/mlp-mixer","category":"vision-transformers","legacyPath":"vision-transformers/2105-mlp-mixer","legacySlug":"2105-mlp-mixer","legacyPaths":[],"title":"MLP-Mixer","subtitle":"少虧就是賺","author":"Z. Yuan","paperDate":"2021-05-04","publishedOn":"2024-07-05","paperTitle":"MLP-Mixer: An all-MLP Architecture for Vision","paperUrl":"https://arxiv.org/abs/2105.01601","bibliography":{"title":"MLP-Mixer: An all-MLP Architecture for Vision","authors":["Ilya Tolstikhin","Neil Houlsby","Alexander Kolesnikov","Lucas Beyer","Xiaohua Zhai","Thomas Unterthiner","Jessica Yung","Andreas Steiner","Daniel Keysers","Jakob Uszkoreit","Mario Lucic","Alexey Dosovitskiy"],"url":"https://arxiv.org/abs/2105.01601","pdfUrl":"https://arxiv.org/pdf/2105.01601","codeUrl":null,"preprint":{"id":"2105.01601","submittedOn":"2021-05-04","url":"https://arxiv.org/abs/2105.01601"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2021/hash/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Abstract.html"},"citation":"@misc{visiontransformersmlpmixer2021,\n  title = {{MLP-Mixer: An all-MLP Architecture for Vision}},\n  author = {Ilya Tolstikhin and Neil Houlsby and Alexander Kolesnikov and Lucas Beyer and Xiaohua Zhai and Thomas Unterthiner and Jessica Yung and Andreas Steiner and Daniel Keysers and Jakob Uszkoreit and Mario Lucic and Alexey Dosovitskiy},\n  year = {2021},\n  eprint = {2105.01601},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2105.01601}\n}\n","citationPath":"/content/papers/citations/vision-transformers/mlp-mixer.81f47c46741e.bib"},"thumbnail":{"src":"/content/papers/assets/9b380c5bd3149092e175b3f465a5061e872e1783fcc2b51ae18dff9d9f274329.jpg","width":1456,"height":772},"locale":"zh-hant","path":"/content/papers/zh-hant/vision-transformers/mlp-mixer.34f174b9e089.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"可視化分析","text":"可視化分析","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/poolformer","relation":"next","topic":"把骨架與 Token 混合方式分開看"}]},{"id":"vision-transformers/caformer","category":"vision-transformers","legacyPath":"vision-transformers/2210-caformer","legacySlug":"2210-caformer","legacyPaths":[],"title":"CAFormer","subtitle":"MetaFormer 使用說明書","author":"Z. Yuan","paperDate":"2022-10-24","publishedOn":"2024-07-11","paperTitle":"MetaFormer Baselines for Vision","paperUrl":"https://doi.org/10.1109/TPAMI.2023.3329173","bibliography":{"title":"MetaFormer Baselines for Vision","authors":["Weihao Yu","Chenyang Si","Pan Zhou","Mi Luo","Yichen Zhou","Jiashi Feng","Shuicheng Yan","Xinchao Wang"],"url":"https://doi.org/10.1109/TPAMI.2023.3329173","pdfUrl":"https://arxiv.org/pdf/2210.13452v4","codeUrl":"https://github.com/sail-sg/metaformer","preprint":{"id":"2210.13452","submittedOn":"2022-10-24","url":"https://arxiv.org/abs/2210.13452"},"publication":{"venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","shortVenue":"IEEE TPAMI","year":2024,"kind":"journal","url":"https://doi.org/10.1109/TPAMI.2023.3329173"},"citation":"@misc{visiontransformerscaformer2022,\n  title = {{MetaFormer Baselines for Vision}},\n  author = {Weihao Yu and Chenyang Si and Pan Zhou and Mi Luo and Yichen Zhou and Jiashi Feng and Shuicheng Yan and Xinchao Wang},\n  year = {2022},\n  eprint = {2210.13452},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2210.13452}\n}\n","citationPath":"/content/papers/citations/vision-transformers/caformer.aa84265d674f.bib"},"thumbnail":{"src":"/content/papers/assets/6ccbfbc74d7ee0be5bc2f37dd6cdd936bf9ed5cf9412cdab0f7dcc2c77a6d9ec.jpg","width":1224,"height":1004},"locale":"zh-hant","path":"/content/papers/zh-hant/vision-transformers/caformer.dba87ff476ce.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"starrelu","text":"StarReLU","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"使用基本的-token-mixer","text":"使用基本的 Token Mixer","depth":3},{"id":"使用常見的-token-mixer","text":"使用常見的 Token Mixer","depth":3},{"id":"消融實驗","text":"消融實驗","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/poolformer","relation":"background","topic":"把骨架與 Token 混合方式分開看"},{"id":"vision-transformers/convmixer","relation":"comparison","topic":"用卷積混合影像切塊"},{"id":"classic-cnns/convnext","relation":"comparison","topic":"用卷積混合影像切塊"}]}],"text":"Transformer 架構帶入視覺領域之後，帶起來一大波的研究風潮。\n\n在過去的研究中，多數人研究如何提升 Token-Mixer 的性能，也有一部分的人研究如何優化 Patchify 的過程。\n\n但是，這些研究都忽略了一個重要的問題：**框架**。\n\n## 定義問題\n\n![metaformer](/content/papers/assets/31f8de4dab18f31e2142924dc8b065e35a2bb63e25082136a06a0ec3b58324a4.jpg)\n\n這個「**框架**」指的組成整個 Transformer 的基本結構，包括：\n\n### Input Embedding\n\n這個部分是將圖片轉換成一個向量序列，這個過程通常是透過 CNN 來完成。\n\n\n> [!TIP]\n> \n> 通常是不重疊滑動視窗的大核卷積，因此雖然說是 CNN，但實際上的功能是全連接層，用 CNN 的原因是可以透過滑動視窗的機制來達到切塊的效果。\n\n\n### Token-Mixer\n\n這個部分是「跨 Token」的混合，這個過程通常是透過 Transformer 的 Self-Attention 來完成。\n\n在最近的研究中，引入 MLP-Mixer 的概念，將 Token-Mixer 的 Self-Attention 換成 MLP，這樣可以減少計算量。\n\n### Channel MLP\n\n這個部分是針對每個 Token 內的「跨維度」的混合。\n\n這個部分大多是透過 MLP 來完成。\n\n### 或許 Token-Mixer 不是重點？\n\n作者試圖探索到底是 Transformer 中基於注意力的 Token-Mixer 對其成功貢獻最大，還是 Transformer 的整體架構（即 MetaFormer）更為重要。\n\n他們通過用簡單的池化操作符替換原本的注意力模組來進行實驗，以驗證 MetaFormer 架構本身在模型效能中的關鍵作用。\n\n## 解決問題\n\n### 模型架構\n\n![poolformer](/content/papers/assets/3cb141e9bb1247410c8214276b21b28ed1d536060d470406f1bfb0c88fe68df2.jpg)\n\n作者為了驗證 MetaFormer 的重要性，弱化了 Token-Mixer 的作用，將原本的 Self-Attention 模組替換成了池化操作。這個操作的簡單程度，大概僅次於恆等映射；PoolFormer 使用的是局部平均池化，讓每個位置聚合鄰近 token 的特徵。\n\n除此之外，作者採用與 PVT 等模型相似的四階段分層架構，像 CNN 模型一樣逐階段降低空間解析度。\n\n- [**PVT: 空間縮減注意力機制**](/blog/papers/vision-transformers/pvt)\n\n在池化操作的部分，作者也留下了實作細節：\n\n```python {20}\nimport torch.nn as nn\n\nclass Pooling(nn.Module):\n\n    def __init__(self, pool_size=3):\n        super().__init__()\n        self.pool = nn.AvgPool2d(\n            pool_size,\n            stride=1,\n            padding=pool_size//2,\n            count_include_pad=False,\n        )\n\n    def forward(self, x):\n        \"\"\"\n        [B, C, H, W] = x.shape\n        Subtraction of the input itself is added since the block already has a\n        residual connection.\n        \"\"\"\n        return self.pool(x) - x\n```\n\n這裡特別注意到實作中的這句 `self.pool(x) - x`，第一次看可能會感到困惑。\n\n作者有留下註解，解釋原因是因為在 Token-Mixer 的結構中有殘差連接的機制，即：\n\n$$\nx = x + \\text{TokenMixer}(x)\n$$\n\n這裡減去 `x`，是讓 Token-Mixer 輸出「鄰域平均值減去當前特徵」；區塊外面的殘差連接仍然保留。若先忽略正規化，簡化式是 $x+(\\operatorname{Pool}(x)-x)=\\operatorname{Pool}(x)$；實際區塊會先正規化再做池化，不能直接說殘差已被取消。\n\n### 模型配置\n\n![config](/content/papers/assets/7d9bb342ec30d0da4e9c56110a318e66aa7903c65a999eb1f26908af8b507510.jpg)\n\n這邊的四階段區塊數採用 1:1:3:1 的比例，例如 S12 配成 2、2、6、2 個區塊；第一階段用 7×7、stride 4 的 patch embedding，後續階段用 3×3、stride 2 降採樣。此外，MLP 的擴充比例為 4。\n\n## 討論\n\n### ImageNet 實驗\n\n![imagenet](/content/papers/assets/0d4f3e7e9363944972dfbed2cb7954070f94faf0460bc330c3d225bbddd7727a.jpg)\n\nPoolFormer 的實驗結果十分驚人。在 ImageNet-1K 表中，S24 以 21M 參數、3.4G MACs 得到 80.3%，高於 DeiT-S 的 79.8% 和 ResMLP-S24 的 79.4%；S36 以 31M 參數、5.0G MACs 得到 81.4%，也略高於 PVT-Medium 的 81.2%。\n\n就只是使用「平均池化」作為 Token-Mixer 的替代，就能在這些比較中取得有競爭力的成績，支持作者對 MetaFormer 架構重要性的判斷。\n\n把上表畫成圖表，可以看到這幾組比較中，PoolFormer 以較少的參數和 MACs 得到較高的準確率；圖中仍有其他模型落在不同的成本與準確率位置。\n\n![imagenet_chart](/content/papers/assets/cd51f95c2e4b9eb182dcf3fb1543c9457d5a0fdffdd4b71cf80704117cef1699.jpg)\n\n### 消融實驗\n\n\n> [!TIP]\n> \n> 基於這篇論文的標題，我們認為這一段是全篇最重要的章節。\n\n\n![ablation](/content/papers/assets/f08cd429d59db7617714f9d76fb2dc03e5c96df4e7b7b6881396f8f0f726bd3a.jpg)\n\n為了進一步驗證 PoolFormer 的有效性，作者以 PoolFormer-S12 為基準，分別替換 Token-Mixer、正規化與啟動函數，或移除區塊元件；這些是各自的消融對照。\n\n1. **Token-Mixer**\n\n   - **恆等映射**\n     - 將池化替換為恆等映射後，模型仍達到 74.3% 的 top-1 準確率，低於原本 S12 的 77.2%；這支持整體架構有貢獻，但不能說 Token-Mixer 沒有作用。\n   - **全域隨機矩陣**\n     - 使用初始化後固定的全域隨機矩陣進行 Token 混合，模型準確率為 75.8%，比恆等映射高 1.5 個百分點；這個變體另有約 21M 個凍結參數，表中的 11.9M 沒有計入。\n   - **深度卷積**\n     - 使用深度卷積替代池化，模型達到 78.1% 的 top-1 準確率，比 PoolFormer-S12 高 0.9 個百分點。\n\n2. **Pooling Size**\n\n   - 測試池大小為 3、5、7，發現效能相似，預設為池大小 3。\n\n3. **修改層標準化（MLN）**\n\n   - 計算標記和通道維度的平均值和方差；在這組消融中，比 Layer Normalization 或 Batch Normalization 分別高 0.7 或 0.8 個百分點。\n   - 去除標準化後，模型性能降至 46.1%。\n\n4. **GELU vs ReLU vs SiLU**\n\n   - 使用 ReLU，性能下降 0.8 個百分點。\n   - 使用 SiLU，性能與 GELU 相同。\n   - 預設使用 GELU。\n\n5. **殘差連接和通道 MLP**\n\n   - 沒有殘差連接或通道 MLP，模型無法收斂，準確度僅為 0.1%/5.7%。\n\n6. **混合模型**\n   - 在底部階段使用池化，頂部階段使用注意力或基於空間 MLP 的混合器。其中底下兩階段池化、頂上兩階段注意力的變體，以 16.5M 參數和 2.5G MACs 達到 81.0% 準確率。\n   - 與 ResMLP-B24 相比，其需要 116M 參數和 23G MACs 才能達到相同精確度。\n\n根據上述的消融實驗，作者提出幾個觀點：\n\n- 在這組消融中，MetaFormer 架構本身對準確率有明顯貢獻。\n- 簡單的池化作為 Token-Mixer 也能達到競爭性能。\n- 修改層標準化（MLN）優於普通 LayerNorm 和 BatchNorm。\n- 殘差連接和 Channel MLP 是不可或缺的組件。\n- 結合池化與其他 Token-Mixer 可能是提高效能的有希望方向。\n\n## 結論\n\n在這篇研究中，作者抽象了原本的 Transformer 為 MetaFormer 的通用架構。\n\n將 Transformer 中的注意力機制抽象化為 Token-Mixer。這種抽象化的創新在於，它不僅僅限於特定的 Token-Mixer，從而提供了一種靈活且強大的框架。\n\n透過將池化這種極為簡單的操作作為 Token-Mixer，作者成功地展示了 PoolFormer 模型在各種視覺任務中的競爭力，從而強有力地支持了「**MetaFormer Is Actually What You Need for Vision**」這一主張。\n\n未來的研究方向之一是檢驗 MetaFormer 架構的廣泛適用性和穩健性。\n\n作者建議探索 PoolFormer 在自然語言處理（NLP）任務中的應用，以驗證 MetaFormer 在 NLP 領域的有效性。\n\n如果能夠證明這一點，MetaFormer 作為通用架構的地位將更加堅固。\n","imageSizes":{"/content/papers/assets/31f8de4dab18f31e2142924dc8b065e35a2bb63e25082136a06a0ec3b58324a4.jpg":{"width":1104,"height":780},"/content/papers/assets/3cb141e9bb1247410c8214276b21b28ed1d536060d470406f1bfb0c88fe68df2.jpg":{"width":1498,"height":548},"/content/papers/assets/7d9bb342ec30d0da4e9c56110a318e66aa7903c65a999eb1f26908af8b507510.jpg":{"width":992,"height":1048},"/content/papers/assets/0d4f3e7e9363944972dfbed2cb7954070f94faf0460bc330c3d225bbddd7727a.jpg":{"width":1506,"height":1046},"/content/papers/assets/cd51f95c2e4b9eb182dcf3fb1543c9457d5a0fdffdd4b71cf80704117cef1699.jpg":{"width":1656,"height":492},"/content/papers/assets/f08cd429d59db7617714f9d76fb2dc03e5c96df4e7b7b6881396f8f0f726bd3a.jpg":{"width":1706,"height":842}}}
