{"notes":[{"id":"multimodality/blip","category":"multimodality","legacyPath":"multimodality/2201-blip","legacySlug":"2201-blip","legacyPaths":[],"title":"BLIP","subtitle":"Synthetic Text Technology","author":"Z. Yuan","paperDate":"2022-01-28","publishedOn":"2024-09-28","paperTitle":"BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation","paperUrl":"https://arxiv.org/abs/2201.12086","bibliography":{"title":"BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation","authors":["Junnan Li","Dongxu Li","Caiming Xiong","Steven Hoi"],"url":"https://arxiv.org/abs/2201.12086","pdfUrl":"https://proceedings.mlr.press/v162/li22n/li22n.pdf","codeUrl":"https://github.com/salesforce/BLIP","preprint":{"id":"2201.12086","submittedOn":"2022-01-28","url":"https://arxiv.org/abs/2201.12086"},"publication":{"venue":"ICML","year":2022,"kind":"conference","url":"https://proceedings.mlr.press/v162/li22n.html"},"citation":"@misc{multimodalityblip2022,\n  title = {{BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation}},\n  author = {Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},\n  year = {2022},\n  eprint = {2201.12086},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.12086}\n}\n","citationPath":"/content/papers/citations/multimodality/blip.5ffabae89440.bib"},"thumbnail":{"src":"/content/papers/assets/322ffdc07e192675b67e86d11d738be51834cd39a07d0979d66e8c27f0e8bd42.jpg","width":1760,"height":706},"locale":"en","path":"/content/papers/en/multimodality/blip.c128d33723b9.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solution","text":"Problem Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"pre-training-objectives","text":"Pre-training Objectives","depth":3},{"id":"capfilt","text":"CapFilt","depth":3},{"id":"model-training-setup","text":"Model Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"capfilts-effectiveness","text":"CapFilt’s Effectiveness","depth":3},{"id":"diversity-in-capfilts-synthetic-text","text":"Diversity in CapFilt’s Synthetic Text","depth":3},{"id":"comparison-with-sota-models","text":"Comparison with SOTA Models","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"solution":"problem-solution","med-three-ways-to-read-the-same-image":"model-architecture","three-pre-training-objectives":"pre-training-objectives","capfilt-write-a-caption-then-let-another-model-check-it":"capfilt","data-and-training-bill":"model-training-setup","results-and-discussion":"discussion","how-much-do-captioner-and-filter-each-contribute":"capfilts-effectiveness","downstream-tasks-keep-the-conditions-in-view":"comparison-with-sota-models"},"guide":null,"relatedReading":[{"id":"multimodality/albef","relation":"comparison","topic":"Aligning and fusing image-text representations"},{"id":"multimodality/meter","relation":"comparison","topic":"Aligning and fusing image-text representations"},{"id":"model-tuning/blip2","relation":"next","topic":"Connecting visual encoders to language models"}]},{"id":"multimodality/albef","category":"multimodality","legacyPath":"multimodality/2107-albef","legacySlug":"2107-albef","legacyPaths":[],"title":"ALBEF","subtitle":"Pursuing the Pure Path","author":"Z. Yuan","paperDate":"2021-07-16","publishedOn":"2024-04-06","paperTitle":"Align before Fuse: Vision and Language Representation Learning with Momentum Distillation","paperUrl":"https://arxiv.org/abs/2107.07651","bibliography":{"title":"Align before Fuse: Vision and Language Representation Learning with Momentum Distillation","authors":["Junnan Li","Ramprasaath R. Selvaraju","Akhilesh Deepak Gotmare","Shafiq Joty","Caiming Xiong","Steven Hoi"],"url":"https://arxiv.org/abs/2107.07651","pdfUrl":"https://proceedings.neurips.cc/paper/2021/file/505259756244493872b7709a8a01b536-Paper.pdf","codeUrl":"https://github.com/salesforce/ALBEF","preprint":{"id":"2107.07651","submittedOn":"2021-07-16","url":"https://arxiv.org/abs/2107.07651"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/505259756244493872b7709a8a01b536-Abstract.html"},"citation":"@misc{multimodalityalbef2021,\n  title = {{Align before Fuse: Vision and Language Representation Learning with Momentum Distillation}},\n  author = {Junnan Li and Ramprasaath R. Selvaraju and Akhilesh Deepak Gotmare and Shafiq Joty and Caiming Xiong and Steven Hoi},\n  year = {2021},\n  eprint = {2107.07651},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2107.07651}\n}\n","citationPath":"/content/papers/citations/multimodality/albef.d03db9353ba8.bib"},"thumbnail":{"src":"/content/papers/assets/2b56a1038c6469d43865692f4ee828fca5bfa45119b1cffbbb305512c58a7e59.jpg","width":1224,"height":600},"locale":"en","path":"/content/papers/en/multimodality/albef.49a7ad3d0898.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"knowledge-distillation","text":"Knowledge Distillation","depth":2},{"id":"basic-principle","text":"Basic Principle","depth":3},{"id":"implementation-steps","text":"Implementation Steps","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"albef-model-design","text":"ALBEF Model Design","depth":3},{"id":"pretraining-objectives","text":"Pretraining Objectives","depth":3},{"id":"momentum-distillation-mod","text":"Momentum Distillation (MoD)","depth":3},{"id":"dataset","text":"Dataset","depth":3},{"id":"technical-details","text":"Technical Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"contribution-of-knowledge-distillation","text":"Contribution of Knowledge Distillation","depth":3},{"id":"comparison-with-clip","text":"Comparison with CLIP","depth":3},{"id":"results-on-various-vl-tasks","text":"Results on Various V+L Tasks","depth":3},{"id":"visualization-results","text":"Visualization Results","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"defining-the-problem":"problem-definition","first-what-does-distillation-mean-here":"knowledge-distillation","solving-the-problem":"solution","align-first-then-fuse":"albef-model-design","three-pretraining-objectives":"pretraining-objectives","momentum-distillation-leave-room-for-another-valid-answer":"momentum-distillation-mod","data-and-implementation":"dataset","experimental-results-and-discussion":"discussion","ablations-which-parts-help":"contribution-of-knowledge-distillation","against-clip-and-align-check-the-protocol-first":"comparison-with-clip","reasoning-tasks-and-the-detector-trade-off":"results-on-various-vl-tasks","do-not-mix-the-heatmap-and-grounding-experiments":"visualization-results"},"guide":null,"relatedReading":[{"id":"multimodality/blip","relation":"comparison","topic":"Aligning and fusing image-text representations"},{"id":"multimodality/meter","relation":"comparison","topic":"Aligning and fusing image-text representations"}]},{"id":"multimodality/meter","category":"multimodality","legacyPath":"multimodality/2111-meter","legacySlug":"2111-meter","legacyPaths":[],"title":"METER","subtitle":"Vibrant Dashboard","author":"Z. Yuan","paperDate":"2021-11-03","publishedOn":"2024-04-06","paperTitle":"An Empirical Study of Training End-to-End Vision-and-Language Transformers","paperUrl":"https://arxiv.org/abs/2111.02387","bibliography":{"title":"An Empirical Study of Training End-to-End Vision-and-Language Transformers","authors":["Zi-Yi Dou","Yichong Xu","Zhe Gan","Jianfeng Wang","Shuohang Wang","Lijuan Wang","Chenguang Zhu","Pengchuan Zhang","Lu Yuan","Nanyun Peng","Zicheng Liu","Michael Zeng"],"url":"https://arxiv.org/abs/2111.02387","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2022/papers/Dou_An_Empirical_Study_of_Training_End-to-End_Vision-and-Language_Transformers_CVPR_2022_paper.pdf","codeUrl":"https://github.com/zdou0830/METER","preprint":{"id":"2111.02387","submittedOn":"2021-11-03","url":"https://arxiv.org/abs/2111.02387"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2022/html/Dou_An_Empirical_Study_of_Training_End-to-End_Vision-and-Language_Transformers_CVPR_2022_paper.html"},"citation":"@misc{multimodalitymeter2021,\n  title = {{An Empirical Study of Training End-to-End Vision-and-Language Transformers}},\n  author = {Zi-Yi Dou and Yichong Xu and Zhe Gan and Jianfeng Wang and Shuohang Wang and Lijuan Wang and Chenguang Zhu and Pengchuan Zhang and Lu Yuan and Nanyun Peng and Zicheng Liu and Michael Zeng},\n  year = {2021},\n  eprint = {2111.02387},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.02387}\n}\n","citationPath":"/content/papers/citations/multimodality/meter.94184d6f116d.bib"},"thumbnail":{"src":"/content/papers/assets/8f9d601a33c85ce330b5e148482a9e3377108e1581f48c0719f4ffb6b052831f.png","width":1468,"height":606},"locale":"en","path":"/content/papers/en/multimodality/meter.a39ec14703e8.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"meter-model-design","text":"METER Model Design","depth":3},{"id":"encoder-only-vs-encoder-decoder","text":"Encoder-Only vs. Encoder-Decoder","depth":3},{"id":"pretraining-strategies","text":"Pretraining Strategies","depth":3},{"id":"dataset","text":"Dataset","depth":3},{"id":"technical-details","text":"Technical Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"evaluating-components-separately","text":"Evaluating Components Separately","depth":3},{"id":"vl-combined-evaluation","text":"V+L Combined Evaluation","depth":3},{"id":"feature-fusion-analysis","text":"Feature Fusion Analysis","depth":3},{"id":"expanding-dataset-scale","text":"Expanding Dataset Scale","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"solution":"solving-the-problem","five-knobs-on-one-workbench":"meter-model-design","default-recipe-and-separately-tested-objectives":"pretraining-strategies","experimental-results-and-discussion":"discussion","compare-components-first-without-vlp-still-includes-training":"evaluating-components-separately","put-them-together-text-vision-and-learning-rate":"vl-combined-evaluation","fusion-and-decoder-does-adding-a-part-always-help":"feature-fusion-analysis","from-4-million-to-14-million-images":"expanding-dataset-scale"},"guide":null,"relatedReading":[{"id":"multimodality/albef","relation":"comparison","topic":"Aligning and fusing image-text representations"},{"id":"multimodality/blip","relation":"comparison","topic":"Aligning and fusing image-text representations"}]},{"id":"model-tuning/blip2","category":"model-tuning","legacyPath":"model-tuning/2301-blip2","legacySlug":"2301-blip2","legacyPaths":[],"title":"BLIP-2","subtitle":"Q-Former debuts","author":"Z. Yuan","paperDate":"2023-01-30","publishedOn":"2024-11-18","paperTitle":"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models","paperUrl":"https://arxiv.org/abs/2301.12597","bibliography":{"title":"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models","authors":["Junnan Li","Dongxu Li","Silvio Savarese","Steven Hoi"],"url":"https://arxiv.org/abs/2301.12597","pdfUrl":"https://arxiv.org/pdf/2301.12597","codeUrl":null,"preprint":{"id":"2301.12597","submittedOn":"2023-01-30","url":"https://arxiv.org/abs/2301.12597"},"publication":null,"citation":"@misc{modeltuningblip22023,\n  title = {{BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models}},\n  author = {Junnan Li and Dongxu Li and Silvio Savarese and Steven Hoi},\n  year = {2023},\n  eprint = {2301.12597},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2301.12597}\n}\n","citationPath":"/content/papers/citations/model-tuning/blip2.ce71ab38cdb4.bib"},"thumbnail":{"src":"/content/papers/assets/f31adfa067aff2dd886466748e3c8b979e7d6bf591136c6bd8116ae711f1d686.jpg","width":1358,"height":502},"locale":"en","path":"/content/papers/en/model-tuning/blip2.743fdf342390.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"phase-1-pre-training","text":"Phase 1 Pre-Training","depth":3},{"id":"phase-2-pre-training","text":"Phase 2 Pre-Training","depth":3},{"id":"pre-training-datasets","text":"Pre-Training Datasets","depth":3},{"id":"pre-training-setup","text":"Pre-Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"zero-shot-vqa","text":"Zero-shot VQA","depth":3},{"id":"finetuned-vqa","text":"Finetuned VQA","depth":3},{"id":"image-captioning","text":"Image Captioning","depth":3},{"id":"image-text-retrieval","text":"Image-Text Retrieval","depth":3},{"id":"image-to-text-generation-showcase","text":"Image-to-Text Generation Showcase","depth":3},{"id":"failure-cases","text":"Failure Cases","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"text-to-image-generation-showcase":"image-to-text-generation-showcase"},"guide":null,"relatedReading":[{"id":"multimodality/blip","relation":"background","topic":"Connecting visual encoders to language models"},{"id":"multimodality/xgen-mm","relation":"next","topic":"Connecting visual encoders to language models"}]}],"text":"By this point, multimodal models had firmly established themselves in our field of view. Looking at ImageNet rankings could make it feel as if multimodal architectures were everywhere, though that impression is not a result established by this paper.\n\n## Problem Definition\n\nThe authors of this paper reflect on recent advancements in multimodal models and identify a few key issues with current approaches:\n\n1. **Poor Quality of Training Data:**\n\n   Aside from CLIP, another well-known model is ALIGN, which is trained on massive but noisy datasets. Although many studies have shown that these noisy datasets can still yield good results, the authors believe that this is a compromise due to the lack of proper data processing methods.\n\n2. **Architecture Determines Fate:**\n\n   The choice of architecture in multimodal models generally falls into one of two categories: encoder-only models or encoder-decoder models.\n\n   Encoder-only models struggle to transfer well to text generation tasks, such as image captioning. On the other hand, encoder-decoder models perform poorly in tasks like text retrieval.\n\nIs there no approach that can tackle both tasks effectively?\n\n## Problem Solution\n\n### Model Architecture\n\n![blip arch](/content/papers/assets/322ffdc07e192675b67e86d11d738be51834cd39a07d0979d66e8c27f0e8bd42.jpg)\n\nThe figure above outlines the BLIP architecture, which can be explained from left to right:\n\nFirst, the authors adopt the ViT architecture (Vision Transformer) as the visual input encoder. A special token `[CLS]` is added to the output sequence to represent the features of the entire image.\n\nTo build a unified model capable of both understanding and generating language, the authors introduce a **Multimodal Mixture of Encoder-Decoder (MED)** model. This model has three primary components:\n\n1. **Text Encoder:** This is the second block from the left in the diagram.\n\n   The text encoder works similarly to BERT, where a `[CLS]` token is attached at the beginning of the text input to summarize the sentence.\n\n   - [**[18.10] BERT: Twelve Layers of Encoders**](/blog/papers/transformers/bert)\n\n2. **Image-Grounded Text Encoder:** This is the third block from the left.\n\n   Visual information is injected by inserting a cross-attention (CA) layer between the self-attention (SA) and feedforward network (FFN) layers in each transformer block.\n\n   A special token `[Encode]` is attached at the end of the text sequence to create a multimodal representation of the image and text.\n\n\n> [!TIP]\n> \n> Special tokens are used to mark different sequences, allowing the model to understand the current task during training.\n\n\n\n> [!TIP]\n> \n> Although this mode cross-attends to image features, its text self-attention is still bidirectional: it is an **encoder** for ITM. The generation decoder switches to causal self-attention.\n\n\n3. **Image-Grounded Text Decoder:**\n\n   The bidirectional self-attention layers in the text encoder are replaced with causal self-attention layers in the text decoder.\n\n   The decoding process begins with the `[Decode]` token, which signals the start of the sequence, and an end token is used to signify the sequence’s completion.\n\n\n> [!TIP]\n> \n> This is also a transformer decoder, but since it’s designed for text generation, self-attention is replaced with causal self-attention.\n\n\n### Pre-training Objectives\n\nWith the architecture in place, the next step is defining the pre-training objectives.\n\nBLIP optimizes for three objectives during pre-training, two of which are understanding tasks, and the third is a generation task:\n\n1. **Image-Text Contrastive Loss (ITC):**\n\n   This loss ensures that matched image-text pairs have similar representations in the feature space, while unmatched pairs are far apart. This helps the model learn to correctly associate related images and text and distinguish unrelated pairs.\n\n   ITC uses unimodal encoders to process images and text separately and introduces a momentum encoder to generate features. Additionally, it softens the labels for potential positive samples in the negative set to improve accuracy in learning the relationship between images and text.\n\n2. **Image-Text Matching Loss (ITM):**\n\n   ITM classifies matched and mismatched image–text pairs using the multimodal `[Encode]` representation and a linear head. The negative pairs are **hard negatives** sampled according to high within-batch contrastive similarity; they are not formed by replacing the image at a fixed 0.5 probability. The classification loss trains the match decision.\n\n3. **Language Modeling Loss (LM):**\n\n   The image-grounded text decoder is enabled to generate textual descriptions of the image. This objective optimizes cross-entropy loss to train the model to maximize the probability of generating the correct text in an autoregressive fashion. Unlike the widely-used masked language modeling (MLM) loss, LM loss empowers the model with generation capabilities, turning visual information into coherent text descriptions. A 0.1 label smoothing technique is applied to improve training.\n\n---\n\nTo improve the efficiency of multi-task learning:\n\n- **The authors share all parameters between the text encoder and decoder except for the self-attention (SA) layers.**\n\nThe main difference between the encoding and decoding tasks lies in the self-attention layer: the encoder uses bidirectional self-attention to construct an overall representation of the input, while the decoder uses causal self-attention to predict the next token based on context.\n\nAdditionally, since the embedding layer, cross-attention layers, and feedforward network serve similar functions in both encoding and decoding, sharing these layers boosts training efficiency and makes better use of parameters in multi-task learning.\n\nTable 3 tests this choice: COCO retrieval TR@1 is **78.4** when all but self-attention is shared, **77.3** when everything is shared, and **78.3** when nothing is shared, with **252M, 224M, and 361M** parameters respectively. The score and parameter conditions belong together.\n\n### CapFilt\n\n![capfilt](/content/papers/assets/6ee36eb2a440795edc923f119658ef9561764fe15f6aa453e680b33b6be541c1.jpg)\n\nAs mentioned earlier, the authors believe that training models on noisy datasets can negatively impact their performance. To address this, they propose a **Captioning and Filtering (CapFilt)** method to improve the quality of text data.\n\nAs shown in the diagram, CapFilt uses two models initialized from the **same pretrained MED** and then fine-tuned separately on annotated COCO data:\n\n1. **Captioner:** This model uses the image-grounded text decoder, fine-tuned with the language modeling objective. It generates one synthetic description (Ts) for each web image.\n\n2. **Filter:** This independently fine-tuned model uses the image-grounded text encoder with ITC and ITM objectives to decide whether a text matches an image.\n\nThe filter removes original web descriptions (Tw) or synthetic descriptions (Ts) predicted as mismatches by ITM. Retained original and synthetic pairs join the annotated pairs in a new dataset; the authors then **pretrain a new BLIP model** on it, rather than merely continuing the current model.\n\n### Model Training Setup\n\nThe model is implemented in PyTorch and pre-trained on two 16-GPU nodes.\n\n- The image transformer is initialized from a ViT pre-trained on ImageNet.\n- The text transformer is initialized from BERTbase.\n\nThe authors explore two ViT variants: ViT-B/16 and ViT-L/16. Unless stated otherwise, results presented for \"BLIP\" refer to the ViT-B model. The model is pre-trained for 20 epochs, with batch sizes of 2880 (ViT-B) and 2400 (ViT-L).\n\nAdamW is used as the optimizer with a weight decay of 0.05. Learning rates are linearly warmed up to 3e-4 (ViT-B) and 2e-4 (ViT-L) before decaying at a rate of 0.85. During pre-training, images are randomly cropped to a resolution of 224×224, and during fine-tuning, the image resolution is increased to 384×384.\n\nThe authors use two manually annotated datasets (COCO and Visual Genome) and three web datasets (Conceptual Captions, Conceptual 12M, SBU captions), totaling about 14 million images in the base setup. Expanded experiments add selected LAION images for about 129 million images; the image count should not be confused with the number of image–text pairs. Generating, filtering, and retraining captions all require additional computation.\n\n## Discussion\n\n### CapFilt’s Effectiveness\n\n![capfilt result](/content/papers/assets/c2829d8c28a9841e2996531a614aa8abcba7c3e390d945cef6c92499f65eea08.jpg)\n\nSince CapFilt is one of the key contributions of this paper, its effectiveness is immediately evaluated.\n\nThe authors compare models pre-trained on different datasets to showcase CapFilt’s impact on downstream tasks, including image-text retrieval and image captioning, evaluated under both fine-tuned and zero-shot settings.\n\nWith the same 14M images and ViT-B backbone, either the captioner or the filter improves all the listed Table 1 metrics over the no-CapFilt row; using both improves them further. For example, fine-tuned COCO text/image retrieval R@1 rises from **78.4/60.7** to **80.6/63.1**. The table mixes fine-tuned retrieval and COCO captioning with zero-shot NoCaps captioning, so these are not all label-free tests.\n\nThe 129M-image and ViT-L experiments show gains from CapFilt against matched data and backbone settings. The gains are not monotone in every column: with a fixed ViT-B main model and 129M images, replacing only the captioner and filter with ViT-L changes COCO retrieval R@1 from **81.9/64.3** to **81.2/64.1**, while some captioning scores improve.\n\nBelow, the authors showcase examples of images and their corresponding generated text, illustrating the captioner’s ability to generate new textual descriptions and the filter’s ability to remove noise from both original and synthetic text.\n\n![capfilt example](/content/papers/assets/8d8965cc654be9f81db1e68f224a86e7710c72e3dad52fa91893cf0da0b6c591.jpg)\n\nGreen text represents text accepted by the filter, while red text indicates rejected text.\n\n### Diversity in CapFilt’s Synthetic Text\n\n![capfilt diversity](/content/papers/assets/ec88223ed93741023aaf25025f36ff34ed40277dcb1e7208f0b9766bc4912e53.jpg)\n\nIn CapFilt, the authors employ **nucleus sampling** to generate synthetic text descriptions.\n\n- [**[19.04] The Curious Case of Neural Text Degeneration**](https://arxiv.org/abs/1904.09751)\n\nNucleus sampling is a probabilistic decoding method where each token is sampled from a subset of tokens whose cumulative probability mass exceeds a threshold $p$ (set to 0.9 in the experiment).\n\nIn Table 2, nucleus sampling is compared with **beam search**. The filter rejects **25%** of nucleus captions versus **19%** of beam captions; that is the filter's rejection fraction, not a hand-checked error rate. Yet COCO retrieval TR@1 rises from **79.6 to 80.6** and NoCaps CIDEr from **103.5 to 105.1**. The authors suggest that more varied descriptions provide new information, while beam search tends toward safer captions. The table does not separately measure surprise or information content.\n\n### Comparison with SOTA Models\n\nThe authors compare BLIP with other state-of-the-art (SOTA) models across various datasets. Here are a few highlights:\n\n- **Image-Text Retrieval**\n\n  ![retrieval](/content/papers/assets/c262a3b16a3d578474bf056cc9c5c1b6aff2bc8411d4201a24950fa1f7f2cf82.jpg)\n\n  BLIP’s image-to-text retrieval (TR) and text-to-image retrieval (IR) capabilities are evaluated on COCO and Flickr30K datasets. The pre-trained models are fine-tuned with ITC and ITM losses. To speed up inference, k candidate pairs are selected based on the similarity of image and text features, which are then re-ranked using their ITM scores. The k values are set to 256 for COCO and 128 for Flickr30K.\n\n  With the same 14M pretraining-image count, BLIP beats ALBEF on COCO 5K test retrieval: TR@1 is **80.6 versus 77.6** and IR@1 **63.1 versus 60.7**. Their mean gain is **2.7 percentage points**, not a 2.7-point gain in every column. The Flickr30K “zero-shot” retrieval table transfers a model **fine-tuned on COCO**, without Flickr30K labels; it is not free of retrieval-task fine-tuning.\n\n- **Image Captioning**\n\n  ![captioning](/content/papers/assets/46e4e38ab889c58055481b10b9173dbaf247b611ddd9a9ee50d1d77bca9f5039.jpg)\n\n  The authors evaluate BLIP’s image captioning capabilities on the NoCaps and COCO datasets. They test models fine-tuned on COCO using LM loss. According to previous literature, they add the prompt “a picture of” at the beginning of each description, which provides a slight performance boost.\n\n  With 14M pretraining images, BLIP's COCO CIDEr is **129.7**, just above VinVL's **129.3**; NoCaps overall CIDEr is **105.1**, above LEMON-base's **100.4** with 12M images. A 129M-image BLIP run approaches several LEMON-base scores at 200M images, but does not exceed it on every metric. The captioning models are fine-tuned on COCO; “zero-shot” NoCaps means no NoCaps-label fine-tuning.\n\n\n> [!TIP]\n> \n> LEMON uses a pretrained object detector and **800×1333** input, while BLIP uses **384×384** input without such a detector. That suggests a different inference-cost profile; comparing speed requires measurements on identical hardware.\n\n\n- **Visual Question Answering (VQA)**\n\n  ![vqa](/content/papers/assets/8dbde265fa7abc514293a91397643043d306cacde550b688eb38623a3ae1d194.jpg)\n\n  VQA requires the model to generate answers based on images and questions. Unlike past studies that often treat VQA as a multi-answer classification task, the authors frame it as an answer generation task, allowing for open-ended VQA. During fine-tuning, the pre-trained model first encodes image-question pairs into multimodal embeddings, which are then fed into the answer decoder.\n\n  The VQA model is fine-tuned using LM loss, with ground-truth answers as targets. In Table 8, 14M BLIP scores **77.62** and ALBEF **76.04** on **test-std**, a **1.58-percentage-point** gap. The paper's prose says 1.64; that is the mean of the test-dev gap (**77.54−75.84 = 1.70**) and test-std gap, rather than the test-std difference alone. A 129M BLIP run scores **78.17**, only **0.03 point** above SimVLM-base's **78.14**; the training data and architecture also differ.\n\n  ![vqa result](/content/papers/assets/d6474e9675f96f16242386a4785b3988c8b6ddfce06be29696cbd7e9f9c37082.jpg)\n\n\n> [!TIP]\n> \n> The paper describes SimVLM as using about 13 times more pretraining data and a larger visual backbone. The SimVLM count is roughly **1.8B image–text pairs**, not 1.8B distinct images, and this cross-model comparison does not isolate CapFilt's effect.\n\n\n## Conclusion\n\nThe authors' CapFilt method tackles noisy web image–text data by generating another description and filtering suspected mismatches. In the reported comparisons, using both steps improves several retrieval and captioning scores; the filter cannot certify that every retained caption is correct.\n\nBLIP also performs well on image–text retrieval, image captioning, and visual question answering. Its useful idea is a model that can both understand and generate, then use those abilities to improve its next training set. The captioner, filter, and fresh training run all carry a cost, and each benchmark still needs its own data, backbone, and fine-tuning conditions read carefully.\n","imageSizes":{"/content/papers/assets/322ffdc07e192675b67e86d11d738be51834cd39a07d0979d66e8c27f0e8bd42.jpg":{"width":1760,"height":706},"/content/papers/assets/6ee36eb2a440795edc923f119658ef9561764fe15f6aa453e680b33b6be541c1.jpg":{"width":1758,"height":590},"/content/papers/assets/c2829d8c28a9841e2996531a614aa8abcba7c3e390d945cef6c92499f65eea08.jpg":{"width":2048,"height":580},"/content/papers/assets/8d8965cc654be9f81db1e68f224a86e7710c72e3dad52fa91893cf0da0b6c591.jpg":{"width":3074,"height":422},"/content/papers/assets/ec88223ed93741023aaf25025f36ff34ed40277dcb1e7208f0b9766bc4912e53.jpg":{"width":2792,"height":444},"/content/papers/assets/c262a3b16a3d578474bf056cc9c5c1b6aff2bc8411d4201a24950fa1f7f2cf82.jpg":{"width":1576,"height":500},"/content/papers/assets/46e4e38ab889c58055481b10b9173dbaf247b611ddd9a9ee50d1d77bca9f5039.jpg":{"width":1940,"height":706},"/content/papers/assets/8dbde265fa7abc514293a91397643043d306cacde550b688eb38623a3ae1d194.jpg":{"width":1224,"height":420},"/content/papers/assets/d6474e9675f96f16242386a4785b3988c8b6ddfce06be29696cbd7e9f9c37082.jpg":{"width":1044,"height":764}}}