{"notes":[{"id":"multimodality/visualbert","category":"multimodality","legacyPath":"multimodality/1908-visualbert","legacySlug":"1908-visualbert","legacyPaths":[],"title":"VisualBERT","subtitle":"序幕上的凝視","author":"Z. Yuan","paperDate":"2019-08-09","publishedOn":"2024-06-04","paperTitle":"VisualBERT: A Simple and Performant Baseline for Vision and Language","paperUrl":"https://arxiv.org/abs/1908.03557","bibliography":{"title":"VisualBERT: A Simple and Performant Baseline for Vision and Language","authors":["Liunian Harold Li","Mark Yatskar","Da Yin","Cho-Jui Hsieh","Kai-Wei Chang"],"url":"https://arxiv.org/abs/1908.03557","pdfUrl":"https://arxiv.org/pdf/1908.03557","codeUrl":null,"preprint":{"id":"1908.03557","submittedOn":"2019-08-09","url":"https://arxiv.org/abs/1908.03557"},"publication":null,"citation":"@misc{multimodalityvisualbert2019,\n  title = {{VisualBERT: A Simple and Performant Baseline for Vision and Language}},\n  author = {Liunian Harold Li and Mark Yatskar and Da Yin and Cho-Jui Hsieh and Kai-Wei Chang},\n  year = {2019},\n  eprint = {1908.03557},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1908.03557}\n}\n","citationPath":"/content/papers/citations/multimodality/visualbert.2232d3e3fc57.bib"},"thumbnail":{"src":"/content/papers/assets/04caeb94d9cca475ea8ab47015986cf93c8e7b5a7c76199f2755c11a6e04f21b.jpg","width":1816,"height":672},"locale":"zh-hant","path":"/content/papers/zh-hant/multimodality/visualbert.11c50b01ad0f.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"預訓練機制","text":"預訓練機制","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"vqa-上的表現","text":"VQA 上的表現","depth":3},{"id":"消融實驗","text":"消融實驗","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"multimodality/vlbert","relation":"comparison","topic":"圖文共用的 Transformer 編碼器"},{"id":"multimodality/vilt","relation":"comparison","topic":"圖文共用的 Transformer 編碼器"}]},{"id":"multimodality/vlbert","category":"multimodality","legacyPath":"multimodality/1908-vlbert","legacySlug":"1908-vlbert","legacyPaths":[],"title":"VL-BERT","subtitle":"序幕下的觀影者","author":"Z. Yuan","paperDate":"2019-08-22","publishedOn":"2024-06-04","paperTitle":"VL-BERT: Pre-training of Generic Visual-Linguistic Representations","paperUrl":"https://arxiv.org/abs/1908.08530","bibliography":{"title":"VL-BERT: Pre-training of Generic Visual-Linguistic Representations","authors":["Weijie Su","Xizhou Zhu","Yue Cao","Bin Li","Lewei Lu","Furu Wei","Jifeng Dai"],"url":"https://arxiv.org/abs/1908.08530","pdfUrl":"https://arxiv.org/pdf/1908.08530","codeUrl":null,"preprint":{"id":"1908.08530","submittedOn":"2019-08-22","url":"https://arxiv.org/abs/1908.08530"},"publication":{"venue":"ICLR","year":2020,"kind":"conference","url":"https://arxiv.org/abs/1908.08530"},"citation":"@misc{multimodalityvlbert2019,\n  title = {{VL-BERT: Pre-training of Generic Visual-Linguistic Representations}},\n  author = {Weijie Su and Xizhou Zhu and Yue Cao and Bin Li and Lewei Lu and Furu Wei and Jifeng Dai},\n  year = {2019},\n  eprint = {1908.08530},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1908.08530}\n}\n","citationPath":"/content/papers/citations/multimodality/vlbert.b671fcd26fda.bib"},"thumbnail":{"src":"/content/papers/assets/d332d0a4275d9f06549b898bb356999e7c27c6af447647ea908b3e997116eb96.jpg","width":1576,"height":688},"locale":"zh-hant","path":"/content/papers/zh-hant/multimodality/vlbert.1f1b44f0950b.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"模型架構","text":"模型架構","depth":3},{"id":"預訓練機制","text":"預訓練機制","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"和其他模型的比較","text":"和其他模型的比較","depth":3},{"id":"模型懂自然語言嗎","text":"模型懂自然語言嗎？","depth":3},{"id":"消融實驗","text":"消融實驗","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"multimodality/visualbert","relation":"comparison","topic":"圖文共用的 Transformer 編碼器"},{"id":"multimodality/vilt","relation":"comparison","topic":"圖文共用的 Transformer 編碼器"}]},{"id":"multimodality/vilt","category":"multimodality","legacyPath":"multimodality/2102-vilt","legacySlug":"2102-vilt","legacyPaths":[],"title":"ViLT","subtitle":"你方唱罷我登場","author":"Z. Yuan","paperDate":"2021-02-05","publishedOn":"2024-04-05","paperTitle":"ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision","paperUrl":"https://arxiv.org/abs/2102.03334","bibliography":{"title":"ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision","authors":["Wonjae Kim","Bokyung Son","Ildoo Kim"],"url":"https://arxiv.org/abs/2102.03334","pdfUrl":"https://arxiv.org/pdf/2102.03334","codeUrl":"https://github.com/dandelin/vilt","preprint":{"id":"2102.03334","submittedOn":"2021-02-05","url":"https://arxiv.org/abs/2102.03334"},"publication":{"venue":"ICML","year":2021,"kind":"conference","url":"https://proceedings.mlr.press/v139/kim21k.html"},"citation":"@misc{multimodalityvilt2021,\n  title = {{ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}},\n  author = {Wonjae Kim and Bokyung Son and Ildoo Kim},\n  year = {2021},\n  eprint = {2102.03334},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.03334}\n}\n","citationPath":"/content/papers/citations/multimodality/vilt.1555ccf87693.bib"},"thumbnail":{"src":"/content/papers/assets/9365d24c758fb8098d2dee6d761d9398a0800b712acb97926f5aac408eea2cce.jpg","width":1224,"height":456},"locale":"zh-hant","path":"/content/papers/zh-hant/multimodality/vilt.957ad9774f7a.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"vilt-設計理念","text":"ViLT 設計理念","depth":3},{"id":"vilt-設計細節","text":"ViLT 設計細節","depth":3},{"id":"預訓練策略","text":"預訓練策略","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"效果表現如何","text":"效果表現如何？","depth":3},{"id":"誰是最主要的貢獻","text":"誰是最主要的貢獻？","depth":3},{"id":"複雜度分析","text":"複雜度分析","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"multimodality/visualbert","relation":"comparison","topic":"圖文共用的 Transformer 編碼器"},{"id":"multimodality/vlbert","relation":"comparison","topic":"圖文共用的 Transformer 編碼器"},{"id":"multimodality/pixelbert","relation":"comparison","topic":"直接從像素學圖文特徵"}]}],"text":"早在 2015 年左右，就有很多跨模態模型嘗試，其中大多都是基於 LSTM 的架構。\n\n時序來到 2017 年，Transformer 在 NLP 領域內大殺四方，隨後注意力機制在多模態領域內也開始大放異彩，語言與視覺的結合再次成為研究的焦點。\n\n這麼熱鬧的事情，怎麼能錯過呢？\n\n作者受此啟發，希望能透過注意力機制捕捉影像中的隱式關係，並認為預先訓練能有效學習這些關係。\n\n## 定義問題\n\n根據前人的研究，作者總結出幾個現階段的問題：\n\n- **統一視覺與語言的模型架構**：目前的許多模型都是針對特定的視覺語言任務而設計的，而缺乏一個可以通用於各種任務的模型。\n- **預訓練的重要性**：BERT 已經告訴我們預訓練很重要，但在多模態領域中，預訓練的重要性還沒有被充分探討。\n- **理解圖像語義的挑戰**：圖像和文字還不能很好的進行關聯。\n\n## 解決問題\n\n### 模型架構\n\n\n\n![VisualBERT](/content/papers/assets/04caeb94d9cca475ea8ab47015986cf93c8e7b5a7c76199f2755c11a6e04f21b.jpg)\n\n\n\nVisualBERT 的核心理念是重用 Transformer 中的自注意力機制，自動對齊輸入文字的元素與圖像中的區域。\n\n這裡是基於 BERT 模型，並增加一組稱為 $F$ 的視覺嵌入，用於表示圖像。每個 $f \\in F$ 對應於圖像中的一個邊界區域，該區域由物體偵測器（Object Detector）導出。\n\n每個視覺嵌入 $f$ 由三個嵌入相加而成：\n\n1. **區域特徵嵌入** ($f_o$)：代表該邊界區域的視覺特徵，由卷積神經網路計算。\n2. **段嵌入** ($f_s$)：標示該嵌入為圖像嵌入（不同於文字嵌入）。\n3. **位置嵌入** ($f_p$)：當輸入中提供了文字與邊界區域的對齊信息時，該嵌入表示對齊位置，並設定為對應對齊文字的位置嵌入之和。\n\n將視覺嵌入與原始文字嵌入一起傳入多層 Transformer，讓模型在兩組輸入之間自動發現有用的對齊關係，並生成新的聯合表示特徵。\n\n\n> [!TIP]\n> \n> 這篇是比較早期的論文，因此在跨模態的處理稍嫌生澀，但仍然是個很好的開始。\n\n\n### 預訓練機制\n\nVisualBERT 的訓練目標是學會處理語言與視覺兩種輸入，因此使用了 COCO 資料集。COCO 包含多組配對數據，每張圖片配有 5 個獨立的標註說明。\n\n訓練分為三個主要階段：\n\n首先是與任務無關的預訓練，模型在 COCO 的 Karpathy 訓練集（約 10 萬張圖片）上進行預訓練，目標是完成兩個與視覺相關的語言模型任務：\n\n1. **遮蔽語言建模**：在文字輸入中隨機遮蔽一些文字，要求模型預測被遮蔽的詞，但對應圖片區域的向量不遮蔽。\n2. **句子-圖像配對預測**：對於 COCO 資料集中的每張圖片，有多個相對應的標註說明。模型接收由兩個標註組成的文字片段，其中一個標註描述圖片，另一個有 50% 機率是相對應的另一個標註，另 50% 機率為隨機選取的標註，模型需區分這兩種情況。\n\n接著是針對特定任務的預訓練，模型在特定任務的數據集上進行預訓練，目標是使模型更好地適應目標領域。在進行下游任務的微調之前，利用該任務的數據集和「遮蔽語言建模與圖像」目標進行額外訓練。\n\n最後是微調，將模型與特定任務的輸入、輸出及目標結合，透過訓練最大化模型在該任務上的表現。\n\n## 討論\n\n### VQA 上的表現\n\n\n\n![VQA](/content/papers/assets/d9c86a015fb40313d5d8572fcffa37cb27106f806da9e079cb98d2e4712fb5ef.jpg)\n\n\n\nVQA 的任務是給定一張圖片和一個問題，模型需正確回答該問題。這裡使用的是 VQA 2.0 資料集，包含來自 COCO 的 100 多萬個關於圖片的問題。\n\n模型被訓練來預測最常見的 3,129 個答案。圖像特徵來自於基於 ResNeXt 的 Faster R-CNN，此模型已在 Visual Genome 預訓練。\n\n實驗結果如上表所示，涵蓋了三類方法：\n\n1. 使用相同的視覺特徵和邊界框數量的基線方法。\n2. 本研究的模型。\n3. 不可比較的方法，包括使用 Visual Genome 的額外問答對、多重偵測器以及模型的集成。\n\nVisualBERT 在可比較的設定下，test-dev 分數為 70.80，優於 Pythia v0.3 的 68.71。\n\n\n> [!TIP]\n> \n> 還有許多其他資料集和任務，例如：VCR 和 $\\text{NLVR}^2$ 等，這裡就不一一列舉了。\n\n\n### 消融實驗\n\n\n\n![ablation](/content/papers/assets/e75fd6306f644773981bc55a17290fb5150d3d66316efebf8a8022a4726af177.jpg)\n\n\n\n作者在 NLVR2 數據集上進行消融實驗，測試了 VisualBERT 的兩種消融模型及四種額外變體。為了簡化計算，所有模型僅使用每張圖像的 36 個特徵（包括完整模型）。\n\n實驗分析了以下四個 VisualBERT 主要組件的貢獻：\n\n1. **C1: 任務無關的預訓練 (Task-agnostic Pre-training)**：\n\n   - 為了研究任務無關預訓練的作用，設計了兩個變體：\n     - 不進行 COCO 任務無關預訓練的模型 (VisualBERT w/o COCO Pre-training)。\n     - 僅用 COCO 中的文字（無圖像）進行預訓練的模型 (VisualBERT w/o Grounded Pre-training)。\n   - 結果顯示，這兩個變體的性能均較差，說明在視覺-語言配對數據上進行預訓練十分重要。\n\n2. **C2: 早期融合 (Early Fusion)**：\n\n   - 包含不進行早期融合的變體 (VisualBERT w/o Early Fusion)，以驗證圖像與文字特徵間早期交互的重要性。實驗結果再度確認多層次的視覺和語言交互對性能有顯著貢獻。\n\n3. **C3: BERT 初始化 (BERT Initialization)**：\n\n   - 所有模型均基於預訓練的 BERT 模型進行初始化。為了理解 BERT 初始化的貢獻，設計了一個隨機初始化參數的變體，並按完整模型的方式進行訓練。結果顯示，僅使用語言預訓練的 BERT 權重對性能確有幫助，但性能下降幅度低於預期，說明模型可能在 COCO 預訓練中學到了許多有用的語言-視覺關聯知識。\n\n4. **C4: 句子-圖像配對預測目標 (Sentence-Image Prediction Objective)**：\n   - 引入了不包含句子-圖像配對預測目標的變體 (VisualBERT w/o Objective 2)，以研究該目標在任務無關預訓練中的影響。結果顯示該目標對性能有正面影響，但作用相對其他組件較小。\n\n實驗結果證實，最重要的設計選擇是**任務無關的預訓練 (C1)** 和**視覺與語言的早期融合 (C2)**。在預訓練階段，包含 COCO 資料以及同時使用圖片與標註說明是最重要的部分。\n\n## 結論\n\nVisualBERT 不僅證明了模型的效能，而且透過其內建的注意力機制，提供了一個可解釋和直觀的方式來理解資訊。\n\n但有一件事情，不論如何都無法迴避：\n\n- 當人們嘗試結合物件偵測的模型時，模型的架構立刻變得非常複雜且難以使用。\n- 過度複雜的設計會抑制模型在實際應用中的潛力，並增加了部署的困難。\n\n因此，將此架構進行優化和簡化絕對應該被視為後續的重要研究方向。\n","imageSizes":{"/content/papers/assets/04caeb94d9cca475ea8ab47015986cf93c8e7b5a7c76199f2755c11a6e04f21b.jpg":{"width":1816,"height":672},"/content/papers/assets/d9c86a015fb40313d5d8572fcffa37cb27106f806da9e079cb98d2e4712fb5ef.jpg":{"width":1120,"height":432},"/content/papers/assets/e75fd6306f644773981bc55a17290fb5150d3d66316efebf8a8022a4726af177.jpg":{"width":818,"height":476}}}
