{"notes":[{"id":"model-tuning/blip2","category":"model-tuning","legacyPath":"model-tuning/2301-blip2","legacySlug":"2301-blip2","legacyPaths":[],"title":"BLIP-2","subtitle":"Q-Former debuts","author":"Z. Yuan","paperDate":"2023-01-30","publishedOn":"2024-11-18","paperTitle":"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models","paperUrl":"https://arxiv.org/abs/2301.12597","bibliography":{"title":"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models","authors":["Junnan Li","Dongxu Li","Silvio Savarese","Steven Hoi"],"url":"https://arxiv.org/abs/2301.12597","pdfUrl":"https://arxiv.org/pdf/2301.12597","codeUrl":null,"preprint":{"id":"2301.12597","submittedOn":"2023-01-30","url":"https://arxiv.org/abs/2301.12597"},"publication":{"venue":"ICML","year":2023,"kind":"conference","url":"https://proceedings.mlr.press/v202/li23q.html"},"citation":"@misc{modeltuningblip22023,\n  title = {{BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models}},\n  author = {Junnan Li and Dongxu Li and Silvio Savarese and Steven Hoi},\n  year = {2023},\n  eprint = {2301.12597},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2301.12597}\n}\n","citationPath":"/content/papers/citations/model-tuning/blip2.ce71ab38cdb4.bib"},"thumbnail":{"src":"/content/papers/assets/f31adfa067aff2dd886466748e3c8b979e7d6bf591136c6bd8116ae711f1d686.jpg","width":1358,"height":502},"locale":"en","path":"/content/papers/en/model-tuning/blip2.743fdf342390.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"phase-1-pre-training","text":"Phase 1 Pre-Training","depth":3},{"id":"phase-2-pre-training","text":"Phase 2 Pre-Training","depth":3},{"id":"pre-training-datasets","text":"Pre-Training Datasets","depth":3},{"id":"pre-training-setup","text":"Pre-Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"zero-shot-vqa","text":"Zero-shot VQA","depth":3},{"id":"finetuned-vqa","text":"Finetuned VQA","depth":3},{"id":"image-captioning","text":"Image Captioning","depth":3},{"id":"image-text-retrieval","text":"Image-Text Retrieval","depth":3},{"id":"image-to-text-generation-showcase","text":"Image-to-Text Generation Showcase","depth":3},{"id":"failure-cases","text":"Failure Cases","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"text-to-image-generation-showcase":"image-to-text-generation-showcase"},"guide":null,"relatedReading":[{"id":"multimodality/blip","relation":"background","topic":"Connecting visual encoders to language models"},{"id":"multimodality/xgen-mm","relation":"next","topic":"Connecting visual encoders to language models"}]},{"id":"multimodality/blip","category":"multimodality","legacyPath":"multimodality/2201-blip","legacySlug":"2201-blip","legacyPaths":[],"title":"BLIP","subtitle":"Synthetic Text Technology","author":"Z. Yuan","paperDate":"2022-01-28","publishedOn":"2024-09-28","paperTitle":"BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation","paperUrl":"https://arxiv.org/abs/2201.12086","bibliography":{"title":"BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation","authors":["Junnan Li","Dongxu Li","Caiming Xiong","Steven Hoi"],"url":"https://arxiv.org/abs/2201.12086","pdfUrl":"https://proceedings.mlr.press/v162/li22n/li22n.pdf","codeUrl":"https://github.com/salesforce/BLIP","preprint":{"id":"2201.12086","submittedOn":"2022-01-28","url":"https://arxiv.org/abs/2201.12086"},"publication":{"venue":"ICML","year":2022,"kind":"conference","url":"https://proceedings.mlr.press/v162/li22n.html"},"citation":"@misc{multimodalityblip2022,\n  title = {{BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation}},\n  author = {Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},\n  year = {2022},\n  eprint = {2201.12086},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.12086}\n}\n","citationPath":"/content/papers/citations/multimodality/blip.5ffabae89440.bib"},"thumbnail":{"src":"/content/papers/assets/322ffdc07e192675b67e86d11d738be51834cd39a07d0979d66e8c27f0e8bd42.jpg","width":1760,"height":706},"locale":"en","path":"/content/papers/en/multimodality/blip.c128d33723b9.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solution","text":"Problem Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"pre-training-objectives","text":"Pre-training Objectives","depth":3},{"id":"capfilt","text":"CapFilt","depth":3},{"id":"model-training-setup","text":"Model Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"capfilts-effectiveness","text":"CapFilt’s Effectiveness","depth":3},{"id":"diversity-in-capfilts-synthetic-text","text":"Diversity in CapFilt’s Synthetic Text","depth":3},{"id":"comparison-with-sota-models","text":"Comparison with SOTA Models","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"solution":"problem-solution","med-three-ways-to-read-the-same-image":"model-architecture","three-pre-training-objectives":"pre-training-objectives","capfilt-write-a-caption-then-let-another-model-check-it":"capfilt","data-and-training-bill":"model-training-setup","results-and-discussion":"discussion","how-much-do-captioner-and-filter-each-contribute":"capfilts-effectiveness","downstream-tasks-keep-the-conditions-in-view":"comparison-with-sota-models"},"guide":null,"relatedReading":[{"id":"multimodality/albef","relation":"comparison","topic":"Aligning and fusing image-text representations"},{"id":"multimodality/meter","relation":"comparison","topic":"Aligning and fusing image-text representations"},{"id":"model-tuning/blip2","relation":"next","topic":"Connecting visual encoders to language models"}]},{"id":"multimodality/xgen-mm","category":"multimodality","legacyPath":"multimodality/2408-xgen-mm","legacySlug":"2408-xgen-mm","legacyPaths":[],"title":"xGen-MM","subtitle":"Also Known As BLIP-3","author":"Z. Yuan","paperDate":"2024-08-16","publishedOn":"2024-11-19","paperTitle":"xGen-MM (BLIP-3): A Family of Open Large Multimodal Models","paperUrl":"https://arxiv.org/abs/2408.08872v1","bibliography":{"title":"xGen-MM (BLIP-3): A Family of Open Large Multimodal Models","authors":["Le Xue","Manli Shu","Anas Awadalla","Jun Wang","An Yan","Senthil Purushwalkam","Honglu Zhou","Viraj Prabhu","Yutong Dai","Michael S Ryoo","Shrikant Kendre","Jieyu Zhang","Can Qin","Shu Zhang","Chia-Chih Chen","Ning Yu","Juntao Tan","Tulika Manoj Awalgaonkar","Shelby Heinecke","Huan Wang","Yejin Choi","Ludwig Schmidt","Zeyuan Chen","Silvio Savarese","Juan Carlos Niebles","Caiming Xiong","Ran Xu"],"url":"https://arxiv.org/abs/2408.08872v1","pdfUrl":"https://arxiv.org/pdf/2408.08872v1","codeUrl":"https://github.com/salesforce/LAVIS/tree/xgen-mm","preprint":{"id":"2408.08872","submittedOn":"2024-08-16","url":"https://arxiv.org/abs/2408.08872v1"},"publication":{"venue":"ICCV Workshops","year":2025,"kind":"workshop","url":"https://doi.org/10.1109/ICCVW69036.2025.00644"},"citation":"@misc{multimodalityxgenmm2024,\n  title = {{xGen-MM (BLIP-3): A Family of Open Large Multimodal Models}},\n  author = {Le Xue and Manli Shu and Anas Awadalla and Jun Wang and An Yan and Senthil Purushwalkam and Honglu Zhou and Viraj Prabhu and Yutong Dai and Michael S Ryoo and Shrikant Kendre and Jieyu Zhang and Can Qin and Shu Zhang and Chia-Chih Chen and Ning Yu and Juntao Tan and Tulika Manoj Awalgaonkar and Shelby Heinecke and Huan Wang and Yejin Choi and Ludwig Schmidt and Zeyuan Chen and Silvio Savarese and Juan Carlos Niebles and Caiming Xiong and Ran Xu},\n  year = {2024},\n  eprint = {2408.08872},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2408.08872v1}\n}\n","citationPath":"/content/papers/citations/multimodality/xgen-mm.add46f5fa4f8.bib"},"thumbnail":{"src":"/content/papers/assets/137dfccb64dad60e27d7ec5e72feaf4d8dfca3e06fce13e74b096523d8c6a8ba.jpg","width":1224,"height":876},"locale":"en","path":"/content/papers/en/multimodality/xgen-mm.e9837902eec7.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solving","text":"Problem Solving","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"training-process","text":"Training Process","depth":3},{"id":"pretraining-datasets","text":"Pretraining Datasets","depth":3},{"id":"supervised-fine-tuning-datasets","text":"Supervised Fine-Tuning Datasets","depth":3},{"id":"post-training-datasets","text":"Post-Training Datasets","depth":3},{"id":"results-and-discussion","text":"Results and Discussion","depth":2},{"id":"pretraining-model-performance","text":"Pretraining Model Performance","depth":3},{"id":"fine-tuned-model-performance","text":"Fine-Tuned Model Performance","depth":3},{"id":"pretraining-data-scale","text":"Pretraining Data Scale","depth":3},{"id":"any-resolution-training-strategy","text":"Any-Resolution Training Strategy","depth":3},{"id":"instruction-aware-design","text":"Instruction-Aware Design","depth":3},{"id":"diversity-of-pure-text-data","text":"Diversity of Pure Text Data","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"discussion":"results-and-discussion"},"guide":null,"relatedReading":[{"id":"model-tuning/blip2","relation":"background","topic":"Connecting visual encoders to language models"},{"id":"multimodality/flamingo","relation":"comparison","topic":"Connecting visual inputs to language models"},{"id":"deepseek/deepseek-vl","relation":"comparison","topic":"Connecting visual inputs to language models"}]}],"text":"This is the second-generation architecture named BLIP.\n\nThe first BLIP tackled a range of multimodal tasks. This second generation focuses on using a relatively small trainable bridge to steer large pretrained models in the direction we want.\n\n\n> [!TIP]\n> \n> For readers unfamiliar with BLIP, you can refer to our previous article:\n> \n> - [**[22.01] BLIP: Synthetic Text Technology**](/blog/papers/multimodality/blip)\n\n\n## Defining the Problem\n\nWe continue to focus on the field of Vision-Language Models (VLM).\n\nCurrent pre-training architectures often rely on large-scale training, which not only requires significant resources but also does not transfer well to every downstream task due to their diverse nature.\n\nIf we could directly leverage pre-trained unimodal LLMs and align multimodal features under frozen parameters, wouldn’t it save us the effort of retraining a multimodal model?\n\nWith this idea in mind, the authors designed a lightweight query transformer, called Q-Former, as a bridge to extract key features from image encoders and convert them into outputs that the language model can understand.\n\nIs this really possible? Let’s dive in and learn!\n\n## Solving the Problem\n\n### Model Architecture\n\n\n\n![q-former](/content/papers/assets/f31adfa067aff2dd886466748e3c8b979e7d6bf591136c6bd8116ae711f1d686.jpg)\n\n\n\nThis architecture diagram might look a bit complex, so let’s break it down step by step:\n\nStarting from the far left, an image is input and processed by a pre-trained image encoder. The parameters of this encoder are frozen, meaning it only extracts features without participating in training.\n\nNext, that large block on the right is the \"Q-Former\" module. Its image and text Transformers share self-attention layers, while the image side has a cross-attention layer in every other block. It takes two inputs:\n\n1. **Learned Query Embeddings**: These parameters are randomly initialized, with a length set to 32 tokens as described in the paper.\n2. **Text Input Describing the Image**: This is similar to common language model inputs, consisting of natural language descriptions.\n\nThose shared self-attention layers are initialized from BERT-base; the added cross-attention layers are randomly initialized. Each of the 32 learned queries has 768 dimensions and extracts visual information through cross-attention. Whether queries and text can attend to each other depends on the task-specific mask below.\n\n\n> [!TIP]\n> \n> In other words, we randomly initialize a set of learnable query tokens and let them extract information from the image features. Whether they can also attend to text depends on the attention masks of the three objectives below: ITC separates the streams, ITG keeps queries from seeing text, and ITM lets them interact. Together, these objectives train Q-Former to align the query features with language.\n\n\n### Phase 1 Pre-Training\n\n\n\n![objectives](/content/papers/assets/a10b3fc669fd3f08e3c6b081e467fb5692290af8a85e89fdca380f8698274446.jpg)\n\n\n\nIn the diagram above, **Q** represents the Query token, and **T** represents the Text token. Masked sections are indicated with darker shading.\n\nDuring Phase 1, Q-Former is combined with a frozen image encoder for image-text pair pre-training, optimizing the following three objective functions:\n\n1. **Image-Text Matching (ITM)**\n\n   This objective teaches the queries and text to make fine-grained image-text matches. Gradients update the shared Q-Former, not just an isolated Q-only parameter group.\n\n   The attention mask design, as shown above, allows queries and text tokens to interact. A two-class linear classifier scores each query output, and the matching score is the average of its logits across all queries. A **hard negative mining strategy** is applied during training to generate more challenging negative samples.\n\n2. **Image-Grounded Text Generation (ITG)**\n\n   The generation loss is computed at text outputs, but its gradients also pass through queries and shared modules, teaching the queries to extract useful visual information.\n\n   In ITG, queries can attend to one another but not to text. Each text token can attend to every query and previous text, preventing leakage from future target words.\n\n\n> [!TIP]\n> \n> Instead of using a `[CLS]` token, a `[DEC]` token is introduced as the start token for the decoding task.\n\n\n3. **Image-Text Contrastive Learning (ITC)**\n\n   Alongside matching and image-grounded generation, we also want the query features explicitly aligned with the text features. All three objectives share Q-Former parameters; they differ in masks and prediction targets.\n\n   Compute each query output’s similarity to the text `[CLS]` representation and use the maximum as the image–text similarity. Other pairs in the batch supply negatives.\n\n   To prevent information leakage, a unimodal self-attention mask is used, ensuring that queries and text tokens cannot directly observe each other.\n\n### Phase 2 Pre-Training\n\n\n\n![pretraining](/content/papers/assets/51012adb35c66939c2c2bac2457d3daf21a01176c26660cd8e2c1ef6f47e15ce.jpg)\n\n\n\n\n> [!TIP]\n> \n> The first phase focuses on aligning image features with textual descriptions, while the second phase involves pre-training for **text generation**.\n\n\nAs shown in the diagram, Q-Former is connected to a frozen large language model (LLM) to leverage its language generation capabilities. During this stage, the following two architectures can be employed for generation:\n\n- **Decoder-Only Architecture**: Pre-trained with a language modeling loss, requiring the LLM to generate text conditioned on visual features.\n- **Encoder-Decoder Architecture**: Pre-trained with a prefix language modeling loss. The input text is split into a prefix and a suffix. The prefix text, along with visual features, is sent to the LLM encoder, while the suffix text serves as the generation target for the LLM decoder.\n\n\n> [!TIP]\n> \n> It is important to note that the **Image Encoder** and **LLM** are frozen in this phase. The goal is to use the visual cues extracted by Q-Former to guide the LLM in generating text.\n\n\nIn this design, the query embeddings $Z$ output by Q-Former are linearly projected to the same dimension as the LLM’s text embeddings. These projected query embeddings are prepended to the input text embeddings, serving as **soft visual prompts**.\n\nSince Q-Former has been pre-trained to extract visual features relevant to language, it acts as an information bottleneck, effectively filtering out irrelevant visual details and reducing the burden on the LLM for vision-language alignment.\n\n\n> [!TIP]\n> \n> For readers interested in the concept of \"Soft Prompts,\" you can refer to our earlier article:\n> \n> - [**[21.04] Soft Prompts: Subtle Whispers of Guidance**](/blog/papers/model-tuning/soft-prompts)\n\n\n### Pre-Training Datasets\n\n- The pre-training mixture follows BLIP and contains **129 million images** from COCO, Visual Genome, CC3M, CC12M, SBU, and **115 million** selected from LAION400M.\n- A **CapFilt-style pipeline** produces captions for web images:\n  - BLIP-large generates ten captions per image.\n  - CLIP ViT-L/14 ranks these synthetic captions **together with the original web caption** by image-text similarity. The top two are kept, and one is sampled at each pre-training step.\n\n\n> [!TIP]\n> \n> CapFilt is the caption-generation method from the first BLIP paper, so I will not repeat its details here.\n\n\n### Pre-Training Setup\n\n- **Pre-Training Steps**:\n\n  - **Feature Learning Stage**: 250,000 steps.\n  - **Generative Learning Stage**: 80,000 steps.\n\n- **Batch Sizes**:\n\n  - **Feature Learning**: 2320 (ViT-L) or 1680 (ViT-g).\n  - **Generative Learning**: 1920 (OPT) or 1520 (FlanT5).\n\n- **Compute Efficiency**:\n\n  - Training on a single 16 × A100 (40G) machine:\n    - **Feature Learning Stage (Largest Model)**: Less than 6 days.\n    - **Generative Learning Stage (Largest Model)**: Less than 3 days.\n\n- **Optimizer and Learning Rate**:\n\n  - **Optimizer**: AdamW ($\\beta_1 = 0.9$, $\\beta_2 = 0.98$, weight decay $0.05$).\n  - **Learning Rate Schedule**: Cosine decay with a peak learning rate of $1 \\times 10^{-4}$ and linear warmup for 2000 steps; the minimum learning rate in the second stage is $5 \\times 10^{-5}$.\n\n- **Image Processing**:\n  - **Image Size**: $224 \\times 224$.\n  - **Augmentation**: Random cropping and horizontal flipping.\n\n## Discussion\n\n\n\n![table1](/content/papers/assets/3502748d152b1dcbf176c69777ffa72bf187efc8e78eecade0f58f663544e925.jpg)\n\n\n\nWithout saying much, let’s first look at the results.\n\nCompared to previous state-of-the-art models, BLIP-2 achieves higher performance while significantly reducing the number of trainable parameters required during VLM pre-training.\n\n### Zero-shot VQA\n\n\n\n![table2](/content/papers/assets/131fcb86157a62dbef31a28c0192e1e28ee147882e6d60db59537230f8d2e59c.jpg)\n\n\n\nThe authors first tested BLIP-2’s zero-shot capability. The table above shows BLIP-2’s performance on various VQA tasks, including **VQAv2**, **GQA**, and **OK-VQA**.\n\n**Evaluation Setup**:\n\n- The prompt for the OPT model was \"`Question: {}` Answer:\".\n- The prompt for the FlanT5 model was \"`Question: {}` Short answer:\".\n- Beam search was used to generate answers, with a beam width of 5, and a length penalty of $-1$ to encourage concise answers.\n\nOn VQAv2 test-dev, the best BLIP-2 configuration scores 65.0 versus Flamingo80B’s 56.3, an 8.7-percentage-point gain. The 54× comparison refers to **10.2B versus 188M trainable parameters** in the paper’s overview, not total model size; the largest BLIP-2 configuration still has about 12.1B total parameters. Flamingo’s GQA entry is blank, leaving GQA without a direct comparison.\n\nFlamingo80B scores higher on OK-VQA. The authors hypothesize that its larger Chinchilla language model provides more relevant knowledge.\n\nFirst-stage representation learning is important. Omitting it substantially worsens zero-shot VQA, and the OPT setup deteriorates during training. The paper calls this catastrophic forgetting, but the LLM weights remain frozen: the directly observed result is degraded task performance with the learned visual prompts, not forgetting caused by updating LLM weights.\n\n### Finetuned VQA\n\n\n\n![finetune](/content/papers/assets/a1a39430875d4c87f8b3f37ea6af380542a94a72206e29234aa9cc7829acbacf.jpg)\n\n\n\nThe authors fine-tuned BLIP-2 on annotated VQA data, freezing the LLM’s parameters and fine-tuning only the Q-Former and image encoder.\n\nThe question tokens also interact with Q-Former’s queries so that it can extract image features relevant to the question. In Table 4, BLIP-2 ViT-g OPT6.7B has the highest VQAv2 result among the open-ended generation models shown; the closed-ended BEIT-3 classifier still scores higher.\n\n### Image Captioning\n\n\n\n![table2](/content/papers/assets/76d3f017d104abd16e095e2b8059aa17b23ec56e306258aeb6c78f966800effc.jpg)\n\n\n\nThe authors evaluated BLIP-2's performance on the image captioning task and compared it with other methods.\n\nFor the image captioning task, the initial prompt was \"`a photo of`,\" and the model was trained using language modeling loss. During fine-tuning, the LLM's parameters were frozen, and only the Q-Former and image encoder parameters were updated.\n\nOn the COCO Karpathy test set, BLIP-2's best CIDEr score is **145.8**, above the other methods in Table 3; OFA has the highest BLEU@4 at **43.9**. After COCO fine-tuning, BLIP-2 also has the table's highest overall CIDEr and SPICE on zero-shot transfer to the NoCaps validation set.\n\n### Image-Text Retrieval\n\n\n\n![retrieval](/content/papers/assets/44087593a931a07fb47ba3e12b964256dda44615c5dd167cdcfbf48a89e50ddd.jpg)\n\n\n\nRetrieval does not require the LLM. The first-stage image encoder and Q-Former are fine-tuned on COCO, evaluated there and transferred zero-shot to Flickr30K. ITC similarity selects 128 candidates, followed by ITM reranking.\n\nBLIP-2 leads the listed methods on most R@1 measures for Flickr30K zero-shot retrieval and COCO fine-tuned retrieval. Other methods lead or tie on some R@5 and R@10 measures.\n\nThe authors then examined the contribution of each loss in the table below. **ITC** and **ITM** directly learn image-text similarity. For the **COCO fine-tuned retrieval objective** in Table 6, adding **ITG** raises image-to-text R@1 from **84.5 to 85.4** and text-to-image R@1 from **67.2 to 68.3**. This supports the idea that ITG helps queries extract text-relevant visual features.\n\n\n\n![retrieval](/content/papers/assets/74f2821063f489ef818c97dc9f53ac790032c4ff4deeff9b9f5edaab6b44f444.jpg)\n\n\n\n### Image-to-Text Generation Showcase\n\n\n\n![demo](/content/papers/assets/60a69ba0565543eb991f5e4a3847a8824c39095b4893b1aea2924061464eee15.jpg)\n\n\n\nThe authors showcased examples of various zero-shot image-to-text capabilities, including visual knowledge reasoning, visual commonsense reasoning, visual conversation, and personalized image-to-text generation.\n\n\n> [!TIP]\n> \n> The images in the paper are quite large. For space considerations, we have not included the entire image here. Interested readers can refer to the original paper.\n\n\n### Failure Cases\n\n\n\n![fail demo](/content/papers/assets/e3cf321fedb8ef7711aa562766237c24b239951a804c16b7411dae05f5c802a8.jpg)\n\n\n\nThe three examples show inaccurate knowledge, an incorrect reasoning path, and outdated information. The model attributes someone else's quote to Einstein, overlooks Canadian winter weather, and mistakes an iPhone 14 for an iPhone 11. These are problems for future work.\n\n## Conclusion\n\nThe idea behind Q-Former is exciting: a small set of query tokens can guide a large language model toward multimodal tasks through a trainable bridge. The 32 queries are only its visual bottleneck; training updates the Q-Former module, which has about 188M parameters, rather than just those vectors. “Lightweight” is relative to the large frozen backbones.\n\nIf we want to address problems in specific domains, this approach allows us to transfer the capabilities of large pre-trained models toward our desired objectives.\n\n\n> [!TIP]\n> \n> In recent years, Q-Former has become a common sight in the field of Face Anti-Spoofing (FAS) research!\n","imageSizes":{"/content/papers/assets/f31adfa067aff2dd886466748e3c8b979e7d6bf591136c6bd8116ae711f1d686.jpg":{"width":1358,"height":502},"/content/papers/assets/a10b3fc669fd3f08e3c6b081e467fb5692290af8a85e89fdca380f8698274446.jpg":{"width":760,"height":412},"/content/papers/assets/51012adb35c66939c2c2bac2457d3daf21a01176c26660cd8e2c1ef6f47e15ce.jpg":{"width":1606,"height":482},"/content/papers/assets/3502748d152b1dcbf176c69777ffa72bf187efc8e78eecade0f58f663544e925.jpg":{"width":1728,"height":376},"/content/papers/assets/131fcb86157a62dbef31a28c0192e1e28ee147882e6d60db59537230f8d2e59c.jpg":{"width":1224,"height":584},"/content/papers/assets/a1a39430875d4c87f8b3f37ea6af380542a94a72206e29234aa9cc7829acbacf.jpg":{"width":1024,"height":720},"/content/papers/assets/76d3f017d104abd16e095e2b8059aa17b23ec56e306258aeb6c78f966800effc.jpg":{"width":1614,"height":504},"/content/papers/assets/44087593a931a07fb47ba3e12b964256dda44615c5dd167cdcfbf48a89e50ddd.jpg":{"width":1556,"height":634},"/content/papers/assets/74f2821063f489ef818c97dc9f53ac790032c4ff4deeff9b9f5edaab6b44f444.jpg":{"width":912,"height":264},"/content/papers/assets/60a69ba0565543eb991f5e4a3847a8824c39095b4893b1aea2924061464eee15.jpg":{"width":1224,"height":788},"/content/papers/assets/e3cf321fedb8ef7711aa562766237c24b239951a804c16b7411dae05f5c802a8.jpg":{"width":1588,"height":526}}}
