Visual Instruction Tuning
- Publication
- NeurIPS 2023Conference
- First submitted to arXiv
- arXiv:2304.08485
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalityllava2023,
title = {{Visual Instruction Tuning}},
author = {Haotian Liu and Chunyuan Li and Qingyang Wu and Yong Jae Lee},
year = {2023},
eprint = {2304.08485},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2304.08485}
}
The GPT-4 technical report that OpenAI published in March 2023 includes a few examples of answering questions about images. One photo shows a man standing at the back of a taxi ironing clothes; asked "What is unusual about this image?", GPT-4 answers spot on. But the image feature wasn't open to the public at the time, so all anyone could see were the screenshots in the report.
On the open-source side, LLaMA had only just been released, and Alpaca and Vicuna had fine-tuned LLaMA on machine-generated instruction data to produce decent chat models. All of these were text-only.
A month later, Liu, Li and colleagues from the University of Wisconsin–Madison, Microsoft Research and Columbia University published LLaVA (Large Language and Vision Assistant), carrying instruction tuning over to image-text multimodality. The most interesting part of the paper is the data. They had GPT-4 help generate training data for "conversations about images," but the GPT-4 they used couldn't see any images; it was the text-only version.
Defining the problem
Before this, vision models were mostly one model per task. Classification, detection, segmentation and image captioning were each trained separately, language was used only to describe image content, and the instructions a user could give were very limited.
Meanwhile, image-text models such as Flamingo and BLIP-2 could already connect images to large language models and did well at image captioning and visual question answering. But they were trained on large amounts of image-text pairs and never specifically practiced "answering according to the user's instruction." Ask one "What is unusual about this image?" and it would mostly just describe the image.
To teach a model to look at images according to instructions, you need data in the form "image + instruction + answer." There are plenty of image-text pairs on the web, from CC to LAION, but image-text instruction data barely existed, writing it by hand takes a lot of time, and nobody had settled what it should look like.
Solving the problem
Questions from a GPT-4 that can't see
The simplest approach is to rewrite image-text pairs directly into instruction format. Take an image and its caption , then randomly pick a question along the lines of "Describe the image concisely":
Human : X_q X_v <STOP>
Assistant : X_c <STOP>
This is cheap, but the questions and answers are monotonous, with no reasoning at all.
The authors instead asked GPT-4 (or ChatGPT), which accepts only text, to act as the teacher. Since it can't see the image, the image is translated into text for it, using two kinds of symbolic representation:
- Captions: every COCO image has several human-written sentences describing the scene from different angles.
- Bounding boxes: each object's category and location, with coordinates normalized to between 0 and 1.
The paper's example is a photo of an underground parking garage:

The top half is everything GPT-4 receives, five captions plus a list of bounding boxes. The photo in the top right is there for the paper's readers; GPT-4 itself never sees it. The bottom half is the three kinds of response it wrote from that text.
GPT-4 is then asked to produce three kinds of data:
- Conversation: a simulated user asks and an assistant answers, in a tone as if the assistant were really looking at the image. Questions cover object types, counts, actions, locations and relative positions, and only ask about things with a definite answer. For example, "What type of vehicle is featured in the image?" "The image features a black sport utility vehicle (SUV)…"
- Detailed description: a question is drawn at random from a pre-compiled list, such as "Describe the following image in detail," and GPT-4 is asked to write a full description, for example "The image is an underground parking area with a black sport utility vehicle (SUV) parked. There are three people in the scene… They are all working together to pack their luggage into the SUV for a trip."
- Complex reasoning: questions that require step-by-step inference, such as "What challenges do these people face?", answered with "…surrounded by various pieces of luggage, including suitcases and backpacks. They are facing the challenge of fitting all their luggage into the black SUV…"
For each kind, the authors first hand-wrote a few examples to serve as GPT-4's few-shot prompt, and that is the only human annotation in the entire data collection process. The result is 158K examples: 58K conversations, 23K detailed descriptions and 77K complex reasoning examples. The authors compared ChatGPT and GPT-4 in early experiments, and GPT-4 consistently produced better data, especially for spatial reasoning.
The teacher can't see the image and has only read text descriptions of it, yet the student has to look at the actual image and answer. We'll come back to this later.
Architecture: one linear projection

The architecture is very simple, with three parts:
- Vision encoder: a pretrained CLIP ViT-L/14, which cuts the image into small patches and outputs one feature vector per patch. The authors use the features from the layer before the last Transformer layer.
- Projection matrix : a single linear layer that turns each image feature into the same dimension as the language model's word embeddings, .
- Language model: Vicuna, a chat model obtained by fine-tuning LLaMA on ChatGPT conversations that users shared on ShareGPT. The authors chose it because, among open-weight models at the time, it was the best at following instructions.
After the projection, the image features look just like the text's word embeddings and can be joined directly and fed into the language model. For a 224×224 image, CLIP ViT-L/14 cuts a patch every 14×14 pixels and yields 256 features of 1024 dimensions; the projection matrix turns them into 256 visual tokens of 5120 dimensions, about 5.25 million parameters including the bias. These 256 tokens are placed in front of the instruction's text tokens. As far as the language model is concerned, they are just a string of "words" with rather special meanings.
The only thing connecting the two sides is this linear layer with 5.25 million parameters. The authors also mention that more elaborate connections could be used, such as Flamingo's gated cross-attention or BLIP-2's Q-Former, but the simple projection let them run data experiments quickly, so they went with it for now.
Training: two stages
Each image's multi-turn conversation forms one sequence. In the first turn, the image is randomly placed before or after the question, and later turns contain only the question; the loss is computed only on the assistant's answers, including the <STOP> that marks where to stop (### in the implementation).
Stage 1: feature alignment. 595K image-text pairs are filtered from CC3M and turned into single-turn conversations with the simple rewriting described above. The vision encoder and language model are both frozen, and only the projection matrix is trained. In the authors' words, this step amounts to training a compatible "visual tokenizer" for the frozen language model. It runs for 1 epoch on 8 A100s and finishes within 4 hours.
Stage 2: end-to-end fine-tuning. The vision encoder stays frozen, and the projection layer and language model are trained together. The chat version is trained on the 158K instruction examples above for 3 epochs, about 10 hours; a ScienceQA version is also trained, with the question and context as the instruction and the reasoning and answer as the response.
Discussion
The man ironing clothes
The authors first tried the image from the GPT-4 report, asking "What is unusual about this image?"

Excerpts from each model's answer:
- LLaVA: "The unusual aspect of this image is a man ironing clothes on the back of a minivan or van. … one would usually iron clothes in a more stationary and safe location, such as a home, using a regular ironing board. … it is not clear how the man is able to maintain balance and stability while ironing clothes in such an unstable environment."
- GPT-4: "The unusual thing about this image is that a man is ironing clothes on an ironing board attached to the roof of a moving taxi."
- BLIP-2: "a man is sitting on the back of a yellow cab"
- OpenFlamingo: "The man is drying his clothes on the hood of his car."
BLIP-2 and OpenFlamingo are only describing the scene, and not very accurately at that. LLaVA answers the question and picks up on the same point as GPT-4.
The authors also started a new conversation and asked only "What's happening in the scene?" After describing the street, LLaVA brought up on its own that the man is attracting attention by ironing his clothes on top of a moving car. The training data has only about 80,000 unique images, and this image isn't among them.
GPT-4 as the judge
A few examples aren't enough, so the authors also designed a quantitative evaluation. LLaVA looks at the image and answers, while the text-only GPT-4 reads the ground-truth text description and answers, serving as an approximate upper bound; then the question, the text description and both answers are handed to GPT-4 for scoring, with 1 to 10 points for helpfulness, relevance, accuracy and level of detail. The reported "relative score" is LLaVA's score divided by GPT-4's.
The first evaluation set takes 30 images from the COCO validation set and generates the three kinds of questions for each image in the same way, 90 questions in total. Comparing different training data:
| Training data | Conv. | Detail | Reasoning | All |
|---|---|---|---|---|
| Full data | 83.1 | 75.3 | 96.5 | 85.1 |
| Detail + reasoning | 81.5 | 73.3 | 90.8 | 81.9 |
| Conv. + 5% detail + 10% reasoning | 81.0 | 68.4 | 91.5 | 80.5 |
| Conv. only | 76.5 | 59.8 | 84.9 | 73.8 |
| No instruction tuning | 22.0 | 24.0 | 18.5 | 21.5 |
Compiled from Table 4 of the paper. Conv. = conversation, Detail = detailed description, Reasoning = complex reasoning.
The model without instruction tuning scores only 21.5, and after tuning it's above 70, a gap of more than 50 points. Conversation data alone gives 73.8; adding a small amount of detailed description and complex reasoning lifts it to 80.5, and even the conversation questions score higher. Using all three kinds of data is best, at 85.1.
The second evaluation set is harder, with 24 images and 60 questions covering indoor and outdoor scenes, memes, paintings and sketches, each image accompanied by a very detailed human-written description:
| Model | Conv. | Detail | Reasoning | All |
|---|---|---|---|---|
| OpenFlamingo | 19.3 | 19.0 | 19.1 | 19.1 |
| BLIP-2 | 54.6 | 29.1 | 32.9 | 38.1 |
| LLaVA | 57.3 | 52.5 | 81.7 | 67.3 |
Compiled from Table 5 of the paper (average of three inference runs).
LLaVA's overall score is 67.3, 29 points above BLIP-2 and 48 points above OpenFlamingo.
In both evaluations, GPT-4 is the judge and also wrote the reference answers, so the scores are only good for comparing models against each other.
What it can't make out
The authors also picked two examples that LLaVA handles poorly:
- Ichiran ramen: asked "What's the name of the restaurant?", getting it right requires recognizing the restaurant and reading Japanese; asked about the side dishes, it might even need to look things up online.
- An open fridge: asked "What is the brand of the blueberry-flavored yogurt?", it has to make out the Fage logo, which requires a high-resolution image and enough knowledge.
The most interesting one is a third question. The fridge has yogurt and a box of strawberries, but no strawberry-flavored yogurt; asked "Is there strawberry-flavored yogurt in the fridge?", LLaVA answers yes. As the authors put it, it sometimes treats the image as a "bag of patches." It sees the yogurt and it sees the strawberries, but doesn't work out how they relate.
This has a lot to do with being trained by a teacher who can't see the image. However good GPT-4's answers are, they rest only on text descriptions and bounding boxes; details that are in the image but not in the description are something the student never gets a chance to learn.
ScienceQA
ScienceQA is 21,000 multiple-choice science questions covering natural, social and language science; some questions come with an image, and some are text only. LLaVA writes its reasoning before its answer and, trained for 12 epochs, reaches 90.92% accuracy, close to the best result at the time, 91.68%.
The text-only GPT-4 with a two-example few-shot prompt reaches 82.69%, and many of the questions it fails are ones where it says the image or plot is missing. The authors tried two combinations:
- GPT-4 as complement: when GPT-4 can't answer, use LLaVA's answer instead, 90.97%, about the same as LLaVA answering alone.
- GPT-4 as judge: when the two give different answers, hand the question and both answers back to GPT-4 and ask it for a final answer, 92.53%, a new best result at the time.
A GPT-4 that can't see images helps even on questions that come with images. The authors' explanation is that some questions with images can actually be answered without looking at the image, and GPT-4 can recognize these and correct some of LLaVA's mistakes.
A few ablations on ScienceQA:
- Using CLIP features from the last layer gives 89.96%, 0.96 lower than the second-to-last layer. The authors suspect the last layer leans toward global, abstract image properties, while the second-to-last layer keeps more local detail.
- For the order of reasoning and answer, this comparison uses last-layer features. Answer first, trained for 12 epochs, gives 89.77%; reasoning first gets to 89.77% in just 6 epochs and barely improves after that. The authors conclude that reasoning first speeds up convergence but does little for the final result.
- Skipping the stage 1 feature alignment and training directly drops it to 85.81%.
- Switching the model from 13B to 7B drops it to 89.84%.
What came after
Half a year later, the same group of authors published LLaVA-1.5. The changes were few: the image resolution went up to 336 pixels, the linear projection was replaced with an MLP, and academic visual question answering data was added, paired with simple response-format prompts. The 13B version used only 1.2 million examples of public data, trained in about a day on a single machine with 8 A100s, and achieved the best results at the time on 11 benchmarks.
The "vision encoder + projection layer + language model" skeleton is easy to pick up, and many later open-source multimodal models kept it; scoring with GPT-4 as the judge also became more and more common in later model evaluations.
Conclusion
Instruction data doesn't have to be written by people, not even multimodal instruction data. LLaVA turned images into text and handed them to GPT-4 to write questions, getting 158K examples, then connected CLIP to Vicuna with a single linear projection. The two training stages together take a little over half a day on 8 A100s, yet its ability to chat about images already holds up against the examples in the GPT-4 report.
The training order in this paper is well worth copying. First freeze both ends and train only the projection layer, so the image features line up with the language model's word embeddings; then unfreeze the language model for instruction tuning. Skip the first step, and ScienceQA loses 5 percentage points.
So is there strawberry-flavored yogurt in the fridge or not? LLaVA says yes, with full confidence. Probably all its teacher ever told it was "There's yogurt in the fridge, and there are strawberries."