Flamingo: a Visual Language Model for Few-Shot Learning
- Publication
- NeurIPS 2022Conference
- First submitted to arXiv
- arXiv:2204.14198
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalityflamingo2022,
title = {{Flamingo: a Visual Language Model for Few-Shot Learning}},
author = {Jean-Baptiste Alayrac and Jeff Donahue and Pauline Luc and Antoine Miech and Iain Barr and Yana Hasson and Karel Lenc and Arthur Mensch and Katie Millican and Malcolm Reynolds and Roman Ring and Eliza Rutherford and Serkan Cabi and Tengda Han and Zhitao Gong and Sina Samangooei and Marianne Monteiro and Jacob Menick and Sebastian Borgeaud and Andrew Brock and Aida Nematzadeh and Sahand Sharifzadeh and Mikolaj Binkowski and Ricardo Barreira and Oriol Vinyals and Andrew Zisserman and Karen Simonyan},
year = {2022},
eprint = {2204.14198},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2204.14198}
}
In earlier research, it was trendy to forcefully piece together model names, often inspired by Sesame Street characters.
Now, the trend is much simpler: just pick a name you like. For instance, this paper is titled Flamingo.
Problem Definition
CLIP, one of the best-known recent architectures, uses contrastive learning to align image and text embeddings and has achieved strong retrieval results. It also supports zero-shot classification, but has no autoregressive text decoder of its own. Used directly for open-ended tasks such as visual question answering or image captioning, it faces a problem: how would it generate the answer?
On the other hand, GPT-3 has achieved remarkable success across various domains. However, GPT-3 does not support image input.
Can we combine pretrained vision and language models, feed them interleaved images or videos and text, and generate a response? Flamingo's output is text, not a newly generated image or video.
Solution
Model Architecture
The figure above illustrates the Flamingo model architecture. The blue components represent pretrained models with frozen parameters, while the violet components are newly added modules that need to be trained from scratch.
Now, let’s follow the data flow to understand how the Flamingo model operates.
First, prepare the input images, as in the example of a dog and a cat. These images are processed through a vision encoder to obtain visual feature vectors.
The authors use NFNet-F6 pretrained with image–text contrastive learning. Encoder quality affects the results; an arbitrary replacement is not an experimentally equivalent substitute. Its features feed the Perceiver Resampler.
Here the Perceiver Resampler uses learned latent queries to turn a variable number of image or video features into a fixed 64 visual tokens per visual input. That reduces the work of later text–image cross-attention. The number 64 is the token count, not the embedding width; these tokens serve as keys and values for the added cross-attention rather than becoming vocabulary tokens in the original language model.
Now let’s look at the other side—the right-hand side of the figure.
The input format consists of interleaved image and text sequences. For example:
- <image> This is a very cute dog. <image> This is
This image-text sequence is fed into the language model, which passes through a cross-attention layer. The authors made some improvements to this module, naming it GATED XATTN-DENSE.
In this process, language features and image features are fused, and the final output is:
- "a very cute cat."
Cross-Attention Layer
Previously, we mentioned the GATED XATTN-DENSE module. Let’s take a closer look at its design.
At first glance, it seems almost identical to the traditional Transformer decoder architecture, right?
The new module contains both a cross-attention layer and a feed-forward layer, each with a tanh gate. The authors even include implementation code beside the diagram to make this easier to follow.
In short, each added branch has a learnable gate parameter initialized at zero. The tanh of that parameter multiplies the branch output before a residual connection adds it back to the language features. Initially the original language model's output is preserved; training can then teach it to use visual information. A gate is not a probability and need not stay between 0 and 1.
Training Datasets and Strategy
Each text position’s cross-attention directly attends to one associated image or video; earlier visual information can propagate indirectly through text self-attention. M3W training subsequences use up to five images, while few-shot evaluation tests up to 32 examples. This is the tested range, not a claim of a hard 32-example capacity limit.
The training datasets included:
- M3W (MultiModal Massive-Web) Dataset: Extracted interleaved image and text data from approximately 43 million web pages.
- ALIGN Dataset: Contains 1.8 billion image-alt-text pairs.
- LTIP (Long Text & Image Pairs) Dataset: 312 million image-text pairs with higher quality and longer textual descriptions.
- VTP (Video & Text Pairs) Dataset: 27 million short video clips (approximately 22 seconds on average) paired with textual descriptions.
The objective function minimizes the weighted negative log-likelihood over multiple datasets:
where represents the -th dataset, and is the weighting coefficient assigned to that dataset.
Discussion
This paper opens with a wealth of experimental results. Let’s explore them together.
Results Presentation
The flamingo at the very beginning is this paper's mascot. These two images show text completion from few-shot prompts and multi-image dialogue. They help us see how the interleaved format works, but they cannot replace the benchmark comparisons below.
Comparison with Other Models
On tasks with comparable earlier zero-shot or few-shot results, Flamingo obtains stronger results with only 4 in-context examples. Not all 16 listed tasks have a comparable earlier result, and RareAct has only a zero-shot row. In general, few-shot scores tend to improve with model size and example count, much like GPT-3, but individual tasks are not monotonic.
Although Flamingo is limited to processing a maximum of 5 images per sequence during training, it can benefit from up to 32 images or videos during inference, highlighting its flexibility.
Fine-Tuning the Model
With more task-labeled data, the authors fine-tune briefly at a low learning rate and unfreeze the vision encoder to handle higher-resolution input. The table shows new best results on 5 tasks where Flamingo's few-shot result had not previously led: VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes. Other columns, including COCO and VisDial, still trail the listed best.
Separately, without task-specific fine-tuning, one set of Flamingo weights with 32 in-context examples exceeds prior fine-tuned results on six benchmarks. This differs from the preceding experiments that update weights using larger labeled datasets.
Ablation Studies
The ablation uses Flamingo-3B, a shorter training schedule and four-shot validation on five DEV benchmarks. The Overall score divides each task score by its contemporary best score, averages the ratios and multiplies by 100. It is not a single accuracy. Paper Table 3 reports:
| Setting | Overall | Score difference from baseline |
|---|---|---|
| Baseline | 70.7 | — |
| Remove M3W | 53.4 | −17.3 |
| Remove image–text pairs | 60.9 | −9.8 |
| Remove tanh gates | 66.5 | −4.2 |
| Cross-attention every fourth layer | 68.8 | −1.9 |
| Use CLIP ViT-L/14 | 64.9 | −5.8 |
| Use NFNet-F0 | 62.7 | −8.0 |
| Train LM from scratch | 57.8 | −12.9 |
| Unfreeze pretrained LM | 62.7 | −8.0 |
The table helps with the numbers, but let's go through what each experiment is testing. These are Overall score differences, not relative percentage decreases.
- Training data composition
- Removing the interleaved image-text M3W data takes Overall from 70.7 to 53.4.
- Removing conventional image-text pairs takes it from 70.7 to 60.9.
- Removing video-text pairs lowers both video-task scores, MSVDQA and VATEX, in the table.
- Gradient accumulation
- Accumulating gradients across datasets performs better than round-robin updates.
- Cold-start tanh gates
- Removing the gates lowers Overall from 70.7 to 66.5; the authors also observe less stable training.
- Conditioning architecture
- GATED XATTN-DENSE outperforms the tested vanilla cross-attention alternatives.
- Compute, memory, and performance
- Adding GATED XATTN-DENSE at every language-model layer performs best, at the highest tested step cost.
- Adding it every fourth layer changes step time from 1.74 to 1.02 seconds while Overall falls from 70.7 to 68.8. The raw timings are clearer than saying only "66% faster"; they depend on hardware and configuration.
- Perceiver Resampler
- Against MLP and Transformer alternatives at a similar parameter budget, the Perceiver Resampler scores higher Overall and takes less time per step in this setup.
- Vision encoder
- NFNet-F6 scores 70.7 Overall, CLIP ViT-L/14 scores 64.9, and NFNet-F0 scores 62.7. The substitutions also change parameter counts and step time.
- Freezing the language model
- Training a language model from scratch lowers Overall from 70.7 to 57.8. Unfreezing the pretrained language model lowers it to 62.7.
Conclusion
Flamingo is a general-purpose model for image and video tasks. It needs few task-specific examples to adapt through an interleaved prompt, and its architecture supports flexible interactions beyond a fixed visual benchmark. "Few" describes the new-task examples; pretraining still requires substantial data and compute.
The authors have successfully combined pretrained large language models with powerful vision models, taking a significant step toward general-purpose visual understanding. This work provides valuable insights and guidance for future research.