Back to paper library
arXiv 2024ICCV Workshops 2025Multimodal learningPaper notes

xGen-MM

Also Known As BLIP-3

Original paper

xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Paper authors

Le Xue · Manli Shu · Anas Awadalla …

Show all 27 authors

Le Xue · Manli Shu · Anas Awadalla · Jun Wang · An Yan · Senthil Purushwalkam · Honglu Zhou · Viraj Prabhu · Yutong Dai · Michael S Ryoo · Shrikant Kendre · Jieyu Zhang · Can Qin · Shu Zhang · Chia-Chih Chen · Ning Yu · Juntao Tan · Tulika Manoj Awalgaonkar · Shelby Heinecke · Huan Wang · Yejin Choi · Ludwig Schmidt · Zeyuan Chen · Silvio Savarese · Juan Carlos Niebles · Caiming Xiong · Ran Xu

Publication
ICCV Workshops 2025Workshop
First submitted to arXiv
arXiv:2408.08872
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityxgenmm2024,
  title = {{xGen-MM (BLIP-3): A Family of Open Large Multimodal Models}},
  author = {Le Xue and Manli Shu and Anas Awadalla and Jun Wang and An Yan and Senthil Purushwalkam and Honglu Zhou and Viraj Prabhu and Yutong Dai and Michael S Ryoo and Shrikant Kendre and Jieyu Zhang and Can Qin and Shu Zhang and Chia-Chih Chen and Ning Yu and Juntao Tan and Tulika Manoj Awalgaonkar and Shelby Heinecke and Huan Wang and Yejin Choi and Ludwig Schmidt and Zeyuan Chen and Silvio Savarese and Juan Carlos Niebles and Caiming Xiong and Ran Xu},
  year = {2024},
  eprint = {2408.08872},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.08872v1}
}
On this page

Coming up with a name for a research paper is no easy task.

If there’s already a successful model in the past, releasing a sequel might draw more attention. However, to achieve even greater success, perhaps a brand-new name is needed.

So, this time, the authors decided, Why not both? They included both the new and the old names in the paper title!

Problem Definition

Let’s build on the discussion of BLIP-2.

The authors argue that Q-Former and its three objectives—ITM, ITC, and ITG—make large-scale training more complicated. BLIP-2 also mainly handles one image at a time, so an interleaved document with multiple images and text is awkward to feed in. The alternating modalities here are images and text, not two kinds of text.

So, how was this "interleaved data format" handled in the past?

For that, we have to revisit another well-known paper: Flamingo.

Flamingo first compresses each image into visual tokens with a Perceiver Resampler, then inserts trainable gated cross-attention layers between the blocks of a frozen language model. Those layers let the text attend to the images in an interleaved input:

Flamingo paper, Figure 3: frozen vision encoder and language model, with a trained Perceiver Resampler and interleaved cross-attention layers.


The authors still find those extra layers throughout Flamingo's language model rather cumbersome. Its Perceiver Resampler, though, catches their eye: like Q-Former, it condenses image information into a small set of query tokens, but makes for a more direct connector here.

Thus, the goals of this paper are:

  1. Replace Q-Former with a Perceiver Resampler that yields a compact set of visual tokens.
  2. Feed visual and text tokens into one decoder in document order, without Flamingo-style cross-attention blocks between language-model layers. Train with next-text-token prediction.

Finally, to improve model performance, the authors introduced a large-scale, high-quality dataset to enhance the foundation of training.

The larger bet is on data and openness. By releasing the data recipe, models, and fine-tuning code together, the authors give others a way to reproduce and inspect the method. Closing the proprietary-model gap remains a research aim; the individual benchmarks below tell us how far this version gets.

Problem Solving

Model Architecture

xGen-MM paper, Figure 2: the ViT is frozen, while the vision-token sampler and phi3-mini are trained; loss applies to text tokens.

The images and text enter one language decoder in their original order. In the diagram, snowflakes mean frozen parameters and flames mean trainable ones. Do not read the snowflake backward: the ViT stays frozen, while phi3-mini and the vision-token sampler are updated.

The main modules of the model are:

  • Vision Transformer (ViT): Responsible for processing image features.
  • Vision Token Sampler (Perceiver Resampler): Compresses the image embeddings.
  • Pretrained Large Language Model (phi3-mini): Serves as the language processing core for multimodal inputs.

Each image passes through the ViT and Perceiver Resampler; text goes through its tokenizer. The resulting tokens are fed to phi3-mini in their original interleaved order to predict the next text token. There is no image-reconstruction target and no three-loss BLIP-2 objective here.

Training Process

The model architecture is simple, but the training process is intricate. It has four broad stages: pretraining, single-image instruction tuning, interleaved multi-image tuning, and post-training. The last stage has a DPO step and a safety-tuning step.

  • Stage 1: Pretraining

    The model predicts the next text token over a dataset mixture. The 4B base model analyzed here sees roughly 100 billion multimodal tokens, with 384×384384 \times 384 images during pretraining. The any-resolution strategy comes later, during fine-tuning.

  • Stage 2: Supervised Fine-Tuning (SFT)

    Public instruction data teaches the model to answer user requests. Any-resolution token sampling splits larger images into crops, encodes and samples each crop separately, and includes a downscaled full-image view. More crops can mean more visual tokens.

  • Interleaved Multi-Image Supervised Fine-Tuning

    The single-image instruction model receives another round of training with multi-image and interleaved image-text instructions. Some single-image examples remain in the mix to protect that ability. It uses the same any-resolution strategy.

  • Stage 3: Post-Training

    This stage focuses on enhancing the model's usefulness while reducing hallucinations and toxicity.

    This phase is divided into two sub-stages:

    • Sub-Stage 1: Direct Preference Optimization (DPO)
      • Uses paired preferences to improve helpfulness and visual faithfulness.
    • Sub-Stage 2: Safety Fine-Tuning
      • Reduces unsafe responses on the paper's specified safety test.

Pretraining Datasets

xGen-MM paper, Figure 3: pretraining sampling proportions; MINT-1T contributes 32.5%, KALE 25%, and OBELICS 17.5%. A source dataset's total size is not the amount consumed in this run.

The pretraining datasets comprise a mixture of multimodal datasets, as shown in the figure above, sampled according to specific proportions. The datasets include:

  • Multimodal Interleaved Dataset Mixture

    • MINT-1T: The source corpus has about one trillion tokens across HTML, PDF, and arXiv subsets, mixed here at 7:5:1. The model did not consume the entire source corpus.
    • OBELICS: Primarily HTML documents, with different sources and preprocessing from MINT-1T.
  • Caption Dataset Mixture

    • BLIP3-KALE: Knowledge-augmented dense image captions.

    • BLIP3-OCR-200M: PaddleOCR extracts text from 200 million high-resolution DataComp-1B images to build captions at several levels of detail. Bounding-box versions exist, but this model uses text-only OCR information.

xGen-MM paper, Figure 4: six OCR-caption granularities. The model uses the version without bounding-box coordinates.

  • BLIP3-GROUNDING-50M: Open-world tagging and detection models generate object and location descriptions for 50 million DataComp-1B images. These generated labels are not individually human-verified ground truth.

xGen-MM paper, Figure 5: examples of coarse and fine-grained object-location captions and boxes.

  • Other Public Datasets: These include Datacomp-1B, CC12M, CC3M, VG, and SBU, among others.

Supervised Fine-Tuning Datasets

The datasets for this phase primarily include:

  • Instruction-Following Dataset

    • Public datasets from various domains, including multimodal dialogue, image captioning, visual question answering, chart/document understanding, and scientific/mathematical data.
    • Text-only instructions are also mixed in to help preserve language ability. The roughly one million public examples refer to the entire instruction mixture, not the text-only subset; training runs for one epoch.
  • Interleaved Multi-Image Instruction Dataset

    • Training begins with a single-image fine-tuned model, adding multi-image/interleaved image-text instruction data.
    • To prevent degradation of single-image capabilities, some single-image data is mixed into the multi-image training data.

Post-Training Datasets

Post-training is divided into two parts:

  1. Direct Preference Optimization (DPO)

    • VLFeedback contains about 80,000 multimodal instructions. Different models generate responses, which GPT-4V scores for helpfulness, visual faithfulness, and ethics; filtering yields 62,600 preference pairs. DPO runs for one epoch with LoRA updating about 2.5% of the LLM backbone weights. A response to a noised image is also used as an extra dispreferred answer to target the model's own hallucinations.
  1. Safety Fine-Tuning

    • VLGuard contributes about 2,000 unsafe examples involving privacy, sensitive topics, deception, and discrimination. Another 5,000 safe instructions are mixed in. Safety tuning runs for three epochs, again updating about 2.5% of the backbone through LoRA.

Results and Discussion

Pretraining Model Performance

xGen-MM paper, Table 1: 0/4/8-shot VQA accuracy and captioning CIDEr for the 4B base model. Zero-shot scores are especially prompt-sensitive.

The authors compared BLIP-3 with other multimodal models across multiple benchmarks, evaluating its few-shot learning capabilities on tasks like image captioning and visual question answering (VQA).

The focus was on zero-shot performance as well as few-shot settings (4-shots and 8-shots).

At 8 shots, the 4B base model scores 66.9 on VQAv2, 55.3 on TextVQA, and 94.0 on TextCaps, versus MM1-3B's 63.6, 44.6, and 88.8. It also exceeds Idefics-9B in those columns. Yet Idefics2-8B reaches 70.3/57.9 on VQAv2/TextVQA, and MM1-7B has higher COCO/NoCaps CIDEr. Model size, training data, and prompts are not fully controlled across systems.

For xGen-MM itself, most columns improve from zero to four to eight examples; TextVQA, for instance, moves 34.0→54.2→55.3. The paper cautions that zero-shot results are sensitive to prompts. One impressive column is not a universal win.

Fine-Tuned Model Performance

xGen-MM paper, Table 2: single-image instruction models. Avg.(all) and Avg.(perc.) cover different columns; rows with missing scores have no overall average.

The authors conducted a comprehensive evaluation of the fine-tuned model on multimodal (image-text) benchmarks, focusing on the following abilities:

  • General VQA: Assessing the model's ability to answer image-related questions.
  • Visual Perception: Testing image understanding and parsing.
  • Domain Knowledge: Performance on datasets related to mathematics and science.
  • OCR Capability: Handling text-rich images (e.g., documents and charts).
  • Hallucination Mitigation: Reducing the generation of false content.

The comparison mainly includes open and closed models around 3B–4B parameters. GPT-4V is listed as a reference, not as another model under 5B parameters.

Under Table 2's common evaluation, xGen-MM-instruct scores 64.8/66.9 on Avg.(all)/Avg.(perc.); the interleave variant scores 65.1/67.3. Its overall single-image average rises slightly, but not every column does: MMMU falls 42.8→41.1 and CVBench-3D 75.4→72.3. Models with missing columns, including GPT-4V, cannot be compared by that overall average.

The multi-image result is in Table 3 of the paper: instruct→interleave improves BLINK 46.6→49.7, QBench-2 52.4→75.1, and Mantis-eval 42.4→56.7. QBench-2 is above the table's GPT-4V score of 73.4; BLINK and Mantis-eval remain below it. That gives the second training round a more concrete payoff.

Post-training has its own tradeoffs. In Table 4, VLGuard attack success goes from 56.6% to 54.9% after DPO and 5.2% after safety tuning; HallusionBench accuracy goes 56.3→57.1→56.6. The safety result is substantial, but hallucination metrics do not improve monotonically at every step.

Pretraining Data Scale

xGen-MM paper, Figure 7: pretraining tokens against 4/8-shot scores on COCO-Caps, TextCaps, and OK-VQA.

The prose discusses 2B–100B multimodal tokens; the plotted points shown above run from 4B to 100B. The curves compare 4/8-shot scores under one data recipe, without plotting the compute cost of each run.

  • 4B→60B: COCO-Caps and TextCaps CIDEr rise most visibly. The third panel is OK-VQA, not TextVQA.
  • 60B→100B: Most lines continue to climb more slowly; 4-shot OK-VQA, for example, moves from roughly 47.2 to 48.9.

The curves show diminishing marginal gains. Calling 60B the optimal investment would also require GPU time, data quality, and other tasks, none of which this plot optimizes.

Any-Resolution Training Strategy

xGen-MM paper, Figure 8(a): sampling ablation on DocVQA, ChartQA, and OCRBench using a simplified SFT mixture.

The authors investigated three different strategies for image downsampling during training:

  • Anyres patch sampling: A high-resolution image is split into crops plus a downscaled global view. Each crop's ViT embeddings are sampled separately by the Perceiver Resampler before concatenation for the LLM.
  • Base resolution: The image is resized, preserving its aspect ratio, to the visual encoder's default size.
  • Anyres fixed sampling: Embeddings from all crops are concatenated first, then jointly compressed to a fixed 128 or 256 visual tokens.

Patch-wise sampling leads on all three document/OCR benchmarks in the plot: DocVQA is roughly 55 versus 35 for base resolution. These ablations use a simplified SFT recipe, so their numbers should not be mixed with the main Table 2 results. The plot does not report the full speed cost of the extra crops and tokens.

Instruction-Aware Design

xGen-MM paper, Figure 8(b): mean scores across Perception, OCR, Science & Math, and Hallucination groups; instruction tokens bring small changes in both directions.

The authors conducted experiments to test whether incorporating textual instruction tokens during the image downsampling process could enhance model performance. This approach is referred to as instruction-aware visual token sampling.

The averages move slightly in both directions, with no consistent gain; the paper does not report a statistical significance test. The authors hypothesize that a Perceiver Resampler made only of cross-attention layers may not reproduce Q-Former's instruction-aware effect. That is a hypothesis, not an isolated causal finding.

Given the minimal performance difference, the authors retained the original Perceiver Resampler design to simplify the model. They suggest that future research could explore improving instruction-aware visual-language connectors.

Diversity of Pure Text Data

xGen-MM paper, Table 8: equal-sized text-only SFT mixtures; math and coding data produce gains and losses across benchmarks.

The authors examined how the diversity of pure text data impacts multimodal model performance, particularly on multimodal benchmarks involving math and programming tasks.

The study compared two types of pure text datasets:

  • Diverse Data: Includes data from multiple domains such as dialogue, mathematics, and programming.
  • Single-Type Data: Includes only general conversational data.

Adding math and code moves MathVista 37.1→38.9 and MMMU 39.1→40.9. But ScienceQA falls 84.8→81.4, MMStar 46.1→45.3, and MME 64.9→64.8. The table does not support a blanket claim that more diverse text improves every domain.

Maintaining language ability is the authors' reason for including pure-text data in visual instruction tuning. Table 8 measures tradeoffs between equal-sized data mixtures on a few multimodal benchmarks; it does not directly measure how much language ability was preserved.

Conclusion

The authors introduce xGen-MM (BLIP-3) as a comprehensive framework for training a family of open large multimodal models. With carefully assembled large-scale dataset mixtures, it shows multimodal in-context learning ability and achieves impressive results on several multimodal benchmarks. The v1 tables discussed here concern the 4B family, and individual larger models or tasks still provide stronger comparisons.

Competition between open and closed models remains intense. By releasing models, curated datasets, and fine-tuning code, the authors hope to give the research community accessible multimodal foundations and encourage further exploration of LMM capabilities. The reported benchmark gains belong to the stated training and evaluation conditions.