{"notes":[{"id":"model-tuning/cocoop","category":"model-tuning","legacyPath":"model-tuning/2203-cocoop","legacySlug":"2203-cocoop","legacyPaths":[],"title":"CoCoOp","subtitle":"先看過影像再說","author":"Z. Yuan","paperDate":"2022-03-10","publishedOn":"2024-11-30","paperTitle":"Conditional Prompt Learning for Vision-Language Models","paperUrl":"https://arxiv.org/abs/2203.05557","bibliography":{"title":"Conditional Prompt Learning for Vision-Language Models","authors":["Kaiyang Zhou","Jingkang Yang","Chen Change Loy","Ziwei Liu"],"url":"https://arxiv.org/abs/2203.05557","pdfUrl":"https://arxiv.org/pdf/2203.05557","codeUrl":null,"preprint":{"id":"2203.05557","submittedOn":"2022-03-10","url":"https://arxiv.org/abs/2203.05557"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2203.05557"},"citation":"@misc{modeltuningcocoop2022,\n  title = {{Conditional Prompt Learning for Vision-Language Models}},\n  author = {Kaiyang Zhou and Jingkang Yang and Chen Change Loy and Ziwei Liu},\n  year = {2022},\n  eprint = {2203.05557},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.05557}\n}\n","citationPath":"/content/papers/citations/model-tuning/cocoop.be720ff7f0f6.bib"},"thumbnail":{"src":"/content/papers/assets/919586e6d93113289ff42da4bc65226ef2ba90f9b74c61fd9047d1aa7d8fb65e.jpg","width":1224,"height":580},"locale":"zh-hant","path":"/content/papers/zh-hant/model-tuning/cocoop.d04924defffb.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"對於未見類別的泛化能力","text":"對於未見類別的泛化能力","depth":3},{"id":"增加參數量誰不會","text":"增加參數量誰不會？","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-tuning/coop","relation":"background","topic":"用提示調整圖文模型"}]},{"id":"model-tuning/coop","category":"model-tuning","legacyPath":"model-tuning/2109-coop","legacySlug":"2109-coop","legacyPaths":[],"title":"CoOp","subtitle":"搖曳的呢喃","author":"Z. Yuan","paperDate":"2021-09-02","publishedOn":"2024-11-20","paperTitle":"Learning to Prompt for Vision-Language Models","paperUrl":"https://arxiv.org/abs/2109.01134","bibliography":{"title":"Learning to Prompt for Vision-Language Models","authors":["Kaiyang Zhou","Jingkang Yang","Chen Change Loy","Ziwei Liu"],"url":"https://arxiv.org/abs/2109.01134","pdfUrl":"https://arxiv.org/pdf/2109.01134","codeUrl":null,"preprint":{"id":"2109.01134","submittedOn":"2021-09-02","url":"https://arxiv.org/abs/2109.01134"},"publication":{"venue":"International Journal of Computer Vision","shortVenue":"IJCV","year":2022,"kind":"journal","url":"https://doi.org/10.1007/s11263-022-01653-1"},"citation":"@misc{modeltuningcoop2021,\n  title = {{Learning to Prompt for Vision-Language Models}},\n  author = {Kaiyang Zhou and Jingkang Yang and Chen Change Loy and Ziwei Liu},\n  year = {2021},\n  eprint = {2109.01134},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2109.01134}\n}\n","citationPath":"/content/papers/citations/model-tuning/coop.27fdb87fd425.bib"},"thumbnail":{"src":"/content/papers/assets/86ecd435ec6fa318673f5754231fe34217c79955e473b560d3ef270366b2aa5d.jpg","width":1224,"height":564},"locale":"zh-hant","path":"/content/papers/zh-hant/model-tuning/coop.7c71d5114e4d.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"複習-clip","text":"複習 CLIP","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"在多個資料集上的效果","text":"在多個資料集上的效果","depth":3},{"id":"uc-和-csc-的比較","text":"UC 和 CSC 的比較","depth":3},{"id":"泛化能力轉移","text":"泛化能力轉移","depth":3},{"id":"需要多長的提示詞","text":"需要多長的提示詞？","depth":3},{"id":"初始化方法","text":"初始化方法","depth":3},{"id":"提示詞的解釋性","text":"提示詞的解釋性","depth":3},{"id":"結論","text":"結論","depth":3}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"multimodality/clip","relation":"background","topic":"用提示調整圖文模型"},{"id":"model-tuning/cocoop","relation":"next","topic":"用提示調整圖文模型"}]}],"text":"作者發表了 CoOp 後，發現了點問題。\n\n用來引導 VLM 的 token 和輸入影像是獨立的，也就是說，這些 token 在完成訓練之後就不會再去考慮影像的資訊。\n\n\n> [!TIP]\n> \n> 如果你還沒讀過 CoOp 的話，可以參考我們之前寫的文章：\n> \n> - [**[21.09] CoOp: 搖曳的呢喃**](/blog/papers/model-tuning/coop)\n\n\n## 定義問題\n\n由於視覺語言模型通常包含數億到數十億參數，全模型微調在資源需求上不切實際，且可能破壞模型的預訓練表示空間。於是在這之前，作者先提出了 CoOp，透過 Prompt learning 的方式，引導模型泛化到新任務上。\n\n但是 CoOp 在學習上下文時，會過度擬合於特定訓練類別，導致在未見類別中準確率顯著下降。這樣的靜態設計缺乏泛化能力，難以處理更廣泛的場景。\n\n一個直觀的想法是：我們可以讓模型在預測時，可以根據輸入的影像資訊，來調整 token 的內容。\n\n## 解決問題\n\n這一篇論文架構和上一代的 CoOp 幾乎一樣，我們就挑一些重要的部分來看。\n\n### 模型架構\n\n\n\n![model arch](/content/papers/assets/919586e6d93113289ff42da4bc65226ef2ba90f9b74c61fd9047d1aa7d8fb65e.jpg)\n\n\n\nCoCoOp 和 CoOp 的差別在於多了一個分支，根據影像特徵產生條件 token，並加到可學習的上下文向量上。\n\n在此之前，影像和文字的分支是獨立運作的，而這個架構則是讓影像的資訊可以影響到文字提示的表示。\n\n\n> [!TIP]\n> \n> 第一代的 CoOp 架構長這樣：\n> \n> \n> \n> ![coop model arch](/content/papers/assets/86ecd435ec6fa318673f5754231fe34217c79955e473b560d3ef270366b2aa5d.jpg)\n> \n> \n> \n\n\nCoCoOp 的方法是引入一個輕量級神經網路（Meta-Net），結合可學習上下文向量生成輸入條件上下文。Meta-Net 為兩層瓶頸結構（Linear-ReLU-Linear），隱藏層將輸入維度縮至原來的 1/16，輸入為影像編碼器的輸出特徵。\n\n條件上下文向量公式如下，其中 $x$ 是影像編碼器的輸出特徵；同一個 Meta-Net 產生的 $\\pi$ 會加到每個上下文向量上：\n\n$$\nv_m(x) = v_m + \\pi, \\quad \\pi = h_\\theta(x)\n$$\n\n提示格式更新為 $t_i(x) = \\{v_1(x), v_2(x), ..., v_M(x), c_i\\}$，預測機率公式更新為：\n\n$$\np(y|x) = \\frac{\\exp(\\text{sim}(x, g(t_y(x)))/\\tau)}{\\sum_{i=1}^{K} \\exp(\\text{sim}(x, g(t_i(x)))/\\tau)}\n$$\n\n在訓練過程中，上下文向量 $\\{v_m\\}_{m=1}^M$ 與 Meta-Net 參數 $\\theta$ 同時進行優化，基礎模型（CLIP）保持凍結。\n\n\n> [!TIP]\n> \n> 我們又輕鬆愉快地看完一篇論文！\n\n\n## 討論\n\n這裡主要看一下 CoCoOp 和 CoOp 的差異。\n\n### 對於未見類別的泛化能力\n\n![generalization](/content/papers/assets/b5cf52d266905ccab20629d2a39c2032803bec823c998e9bf8ee893b2a741bee.jpg)\n\n作者同樣在 11 個不同的資料集上進行實驗，在表格中，`Base` 表示基礎類別，`New` 表示未見類別，`H` 是兩者的調和平均。`CoOp` 是第一代的模型，`CoCoOp` 是這一篇的模型。\n\n從上表中我們可以觀察到，雖然 CoOp 在基類別中的表現優於 CLIP，但在新類別的準確率顯著下降，平均準確率從基類別的 82.69% 下滑至新類別的 63.22%，相差近 20 個百分點。這可能是因為 CoOp 過度優化基類別，導致新類別表現不佳，這凸顯了提升學習型提示泛化能力的必要性。\n\nCoCoOp 在新類別中的準確率提升至 71.69%，顯著縮小與手工提示間的差距。在 11 個數據集中，有 5 個數據集的新類別準確率比 CoOp 提升超過 10 個百分點；ImageNet 則從 67.88% 提升至 70.43%，甚至超越 CLIP 的 68.14%。\n\n相較於 CoOp，CoCoOp 的基類別準確率在部分數據集中略有下降，但 9 個下降案例中有 6 個數據集的降幅低於 3 個百分點，且新類別的顯著提升足以彌補基類別的損失。例如 StanfordCars 在基類別準確率下降 7.63 個百分點的情況下，新類別準確率提升 13.19 個百分點；兩者相加為正的 5.56 個百分點，表中的調和平均值則從 68.13% 提升至 72.01%。\n\n### 增加參數量誰不會？\n\n\n\n![coop model arch](/content/papers/assets/a1b74219eda2bb6859808b7e97b4aed25fe5b2fff07bd2ed5cabeb8448c277e8.jpg)\n\n\n\nCoCoOp 引入 Meta-Net，增加了參數量，專業的你立刻就會問：\n\n- **這個提升是否僅來自於增加的參數量？**\n\n為此作者設計了一個實驗，移除 CoCoOp 的 Meta-Net，將 CoOp 的上下文數量增加至與 CoCoOp 參數量相近的最大值。\n\n根據上表結果顯示，比較 `CoOp (ctx=60)` 和 `CoOp (ctx=4) + Meta-Net` 在接近參數量的情況下，`CoOp (ctx=60)` 並沒有提升對於新類別的泛化能力；相較於原本的 `CoOp (ctx=4)`，新類別準確率反而降低了 2.54 個百分點。這支持了作者對於 CoOp 過度擬合基礎類別的觀點。\n\n因此，至少在這組 ImageNet 實驗中，單純增加 CoOp 的上下文數量，沒有得到 CoCoOp 的新類別提升。\n\n## 結論\n\nCoCoOp 基於條件提示的設計，解決靜態提示的泛化問題，並且在多種場景中表現卓越，像是跨數據集的提示遷移、域泛化場景等。\n\n作者在最後提到，CoCoOp 的訓練效率顯著低於 CoOp，而且在 11 個數據集中，有 7 個數據集的未見類別性能仍低於 CLIP，顯示學習型提示與手工提示的性能差距尚未完全彌合，問題遠遠還沒到結束的時候。\n\n\n> [!TIP]\n> \n> 在 CLIP 的影像分支和文字分支的中間，加入更多華麗的 Prompt learning 的工程，可說是這幾年的熱門研究方向之一，有時間我們再來彙整一下。\n","imageSizes":{"/content/papers/assets/919586e6d93113289ff42da4bc65226ef2ba90f9b74c61fd9047d1aa7d8fb65e.jpg":{"width":1224,"height":580},"/content/papers/assets/86ecd435ec6fa318673f5754231fe34217c79955e473b560d3ef270366b2aa5d.jpg":{"width":1224,"height":564},"/content/papers/assets/b5cf52d266905ccab20629d2a39c2032803bec823c998e9bf8ee893b2a741bee.jpg":{"width":1224,"height":822},"/content/papers/assets/a1b74219eda2bb6859808b7e97b4aed25fe5b2fff07bd2ed5cabeb8448c277e8.jpg":{"width":784,"height":224}}}
