Back to paper library
arXiv 2023NeurIPS 2023Multimodal learningPaper notes

LLaVA

The teacher who can't see

Original paper

Visual Instruction Tuning

Paper authors

Haotian Liu · Chunyuan Li · Qingyang Wu · Yong Jae Lee

Publication
NeurIPS 2023Conference
First submitted to arXiv
arXiv:2304.08485
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityllava2023,
  title = {{Visual Instruction Tuning}},
  author = {Haotian Liu and Chunyuan Li and Qingyang Wu and Yong Jae Lee},
  year = {2023},
  eprint = {2304.08485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.08485}
}
On this page

The GPT-4 technical report that OpenAI published in March 2023 includes a few examples of answering questions about images. One photo shows a man standing at the back of a taxi ironing clothes; asked "What is unusual about this image?", GPT-4 answers spot on. But the image feature wasn't open to the public at the time, so all anyone could see were the screenshots in the report.

On the open-source side, LLaMA had only just been released, and Alpaca and Vicuna had fine-tuned LLaMA on machine-generated instruction data to produce decent chat models. All of these were text-only.

A month later, Liu, Li and colleagues from the University of Wisconsin–Madison, Microsoft Research and Columbia University published LLaVA (Large Language and Vision Assistant), carrying instruction tuning over to image-text multimodality. The most interesting part of the paper is the data. They had GPT-4 help generate training data for "conversations about images," but the GPT-4 they used couldn't see any images; it was the text-only version.

Defining the problem

Before this, vision models were mostly one model per task. Classification, detection, segmentation and image captioning were each trained separately, language was used only to describe image content, and the instructions a user could give were very limited.

Meanwhile, image-text models such as Flamingo and BLIP-2 could already connect images to large language models and did well at image captioning and visual question answering. But they were trained on large amounts of image-text pairs and never specifically practiced "answering according to the user's instruction." Ask one "What is unusual about this image?" and it would mostly just describe the image.

To teach a model to look at images according to instructions, you need data in the form "image + instruction + answer." There are plenty of image-text pairs on the web, from CC to LAION, but image-text instruction data barely existed, writing it by hand takes a lot of time, and nobody had settled what it should look like.

Solving the problem

Questions from a GPT-4 that can't see

The simplest approach is to rewrite image-text pairs directly into instruction format. Take an image Xv\mathbf{X}_v and its caption Xc\mathbf{X}_c, then randomly pick a question Xq\mathbf{X}_q along the lines of "Describe the image concisely":

text
Human : X_q X_v <STOP>
Assistant : X_c <STOP>

This is cheap, but the questions and answers are monotonous, with no reasoning at all.

The authors instead asked GPT-4 (or ChatGPT), which accepts only text, to act as the teacher. Since it can't see the image, the image is translated into text for it, using two kinds of symbolic representation:

  • Captions: every COCO image has several human-written sentences describing the scene from different angles.
  • Bounding boxes: each object's category and location, with coordinates normalized to between 0 and 1.

The paper's example is a photo of an underground parking garage:

Original Table 1. An example of instruction data for a photo of an underground parking garage. The top half is what is given to the text-only GPT-4, including five COCO captions and the bounding-box coordinates of objects such as person, backpack and suitcase; the photo in the top right is shown only for the reader's reference. The bottom half is the three kinds of response GPT-4 wrote from this: a conversation (what vehicle is in the image), a detailed description (three people in an underground parking garage are loading luggage into a black SUV), and complex reasoning (they have to fit many suitcases and backpacks into this SUV).
Original Table 1, Liu et al., from PDF page 3, scholarly quotation; the photo is an image from the COCO dataset.

The top half is everything GPT-4 receives, five captions plus a list of bounding boxes. The photo in the top right is there for the paper's readers; GPT-4 itself never sees it. The bottom half is the three kinds of response it wrote from that text.

GPT-4 is then asked to produce three kinds of data:

  • Conversation: a simulated user asks and an assistant answers, in a tone as if the assistant were really looking at the image. Questions cover object types, counts, actions, locations and relative positions, and only ask about things with a definite answer. For example, "What type of vehicle is featured in the image?" "The image features a black sport utility vehicle (SUV)…"
  • Detailed description: a question is drawn at random from a pre-compiled list, such as "Describe the following image in detail," and GPT-4 is asked to write a full description, for example "The image is an underground parking area with a black sport utility vehicle (SUV) parked. There are three people in the scene… They are all working together to pack their luggage into the SUV for a trip."
  • Complex reasoning: questions that require step-by-step inference, such as "What challenges do these people face?", answered with "…surrounded by various pieces of luggage, including suitcases and backpacks. They are facing the challenge of fitting all their luggage into the black SUV…"

For each kind, the authors first hand-wrote a few examples to serve as GPT-4's few-shot prompt, and that is the only human annotation in the entire data collection process. The result is 158K examples: 58K conversations, 23K detailed descriptions and 77K complex reasoning examples. The authors compared ChatGPT and GPT-4 in early experiments, and GPT-4 consistently produced better data, especially for spatial reasoning.

The teacher can't see the image and has only read text descriptions of it, yet the student has to look at the actual image and answer. We'll come back to this later.

Architecture: one linear projection

Original Figure 1. LLaVA's architecture. The image X_v passes through the vision encoder to get features Z_v, which the projection matrix W turns into visual tokens H_v; the language instruction X_q is turned into text tokens H_q. Both are fed together into the language model f_φ, which produces the answer X_a.
Original Figure 1, Liu et al., from PDF page 4, CC BY 4.0.

The architecture is very simple, with three parts:

  1. Vision encoder: a pretrained CLIP ViT-L/14, which cuts the image into small patches and outputs one feature vector Zv\mathbf{Z}_v per patch. The authors use the features from the layer before the last Transformer layer.
  2. Projection matrix W\mathbf{W}: a single linear layer that turns each image feature into the same dimension as the language model's word embeddings, Hv=W⋅Zv\mathbf{H}_v = \mathbf{W}\cdot\mathbf{Z}_v.
  3. Language model: Vicuna, a chat model obtained by fine-tuning LLaMA on ChatGPT conversations that users shared on ShareGPT. The authors chose it because, among open-weight models at the time, it was the best at following instructions.

After the projection, the image features look just like the text's word embeddings and can be joined directly and fed into the language model. For a 224×224 image, CLIP ViT-L/14 cuts a patch every 14×14 pixels and yields 256 features of 1024 dimensions; the projection matrix turns them into 256 visual tokens of 5120 dimensions, about 5.25 million parameters including the bias. These 256 tokens are placed in front of the instruction's text tokens. As far as the language model is concerned, they are just a string of "words" with rather special meanings.

The only thing connecting the two sides is this linear layer with 5.25 million parameters. The authors also mention that more elaborate connections could be used, such as Flamingo's gated cross-attention or BLIP-2's Q-Former, but the simple projection let them run data experiments quickly, so they went with it for now.

Training: two stages

Each image's multi-turn conversation forms one sequence. In the first turn, the image is randomly placed before or after the question, and later turns contain only the question; the loss is computed only on the assistant's answers, including the <STOP> that marks where to stop (### in the implementation).

Stage 1: feature alignment. 595K image-text pairs are filtered from CC3M and turned into single-turn conversations with the simple rewriting described above. The vision encoder and language model are both frozen, and only the projection matrix W\mathbf{W} is trained. In the authors' words, this step amounts to training a compatible "visual tokenizer" for the frozen language model. It runs for 1 epoch on 8 A100s and finishes within 4 hours.

Stage 2: end-to-end fine-tuning. The vision encoder stays frozen, and the projection layer and language model are trained together. The chat version is trained on the 158K instruction examples above for 3 epochs, about 10 hours; a ScienceQA version is also trained, with the question and context as the instruction and the reasoning and answer as the response.

Discussion

The man ironing clothes

The authors first tried the image from the GPT-4 report, asking "What is unusual about this image?"

Original Table 3. A photo of a man ironing clothes at the back of a moving yellow taxi, and each model's answer to "What is unusual about this image?" LLaVA points out that someone is ironing clothes on the back of a van, which is both unsafe and unusual; in a new conversation asked only what is happening, it also mentions that he is attracting attention by ironing on a moving vehicle. GPT-4 points out that the ironing board is attached to the roof of a moving taxi; BLIP-2 answers that a man is sitting on the back of a yellow cab; OpenFlamingo answers that he is drying his clothes on the hood.
Original Table 3, Liu et al., from PDF page 6, scholarly quotation; photo source as credited in the paper.

Excerpts from each model's answer:

  • LLaVA: "The unusual aspect of this image is a man ironing clothes on the back of a minivan or van. … one would usually iron clothes in a more stationary and safe location, such as a home, using a regular ironing board. … it is not clear how the man is able to maintain balance and stability while ironing clothes in such an unstable environment."
  • GPT-4: "The unusual thing about this image is that a man is ironing clothes on an ironing board attached to the roof of a moving taxi."
  • BLIP-2: "a man is sitting on the back of a yellow cab"
  • OpenFlamingo: "The man is drying his clothes on the hood of his car."

BLIP-2 and OpenFlamingo are only describing the scene, and not very accurately at that. LLaVA answers the question and picks up on the same point as GPT-4.

The authors also started a new conversation and asked only "What's happening in the scene?" After describing the street, LLaVA brought up on its own that the man is attracting attention by ironing his clothes on top of a moving car. The training data has only about 80,000 unique images, and this image isn't among them.

GPT-4 as the judge

A few examples aren't enough, so the authors also designed a quantitative evaluation. LLaVA looks at the image and answers, while the text-only GPT-4 reads the ground-truth text description and answers, serving as an approximate upper bound; then the question, the text description and both answers are handed to GPT-4 for scoring, with 1 to 10 points for helpfulness, relevance, accuracy and level of detail. The reported "relative score" is LLaVA's score divided by GPT-4's.

The first evaluation set takes 30 images from the COCO validation set and generates the three kinds of questions for each image in the same way, 90 questions in total. Comparing different training data:

Training dataConv.DetailReasoningAll
Full data83.175.396.585.1
Detail + reasoning81.573.390.881.9
Conv. + 5% detail + 10% reasoning81.068.491.580.5
Conv. only76.559.884.973.8
No instruction tuning22.024.018.521.5

Compiled from Table 4 of the paper. Conv. = conversation, Detail = detailed description, Reasoning = complex reasoning.

The model without instruction tuning scores only 21.5, and after tuning it's above 70, a gap of more than 50 points. Conversation data alone gives 73.8; adding a small amount of detailed description and complex reasoning lifts it to 80.5, and even the conversation questions score higher. Using all three kinds of data is best, at 85.1.

The second evaluation set is harder, with 24 images and 60 questions covering indoor and outdoor scenes, memes, paintings and sketches, each image accompanied by a very detailed human-written description:

ModelConv.DetailReasoningAll
OpenFlamingo19.319.019.119.1
BLIP-254.629.132.938.1
LLaVA57.352.581.767.3

Compiled from Table 5 of the paper (average of three inference runs).

LLaVA's overall score is 67.3, 29 points above BLIP-2 and 48 points above OpenFlamingo.

In both evaluations, GPT-4 is the judge and also wrote the reference answers, so the scores are only good for comparing models against each other.

What it can't make out

The authors also picked two examples that LLaVA handles poorly:

  • Ichiran ramen: asked "What's the name of the restaurant?", getting it right requires recognizing the restaurant and reading Japanese; asked about the side dishes, it might even need to look things up online.
  • An open fridge: asked "What is the brand of the blueberry-flavored yogurt?", it has to make out the Fage logo, which requires a high-resolution image and enough knowledge.

The most interesting one is a third question. The fridge has yogurt and a box of strawberries, but no strawberry-flavored yogurt; asked "Is there strawberry-flavored yogurt in the fridge?", LLaVA answers yes. As the authors put it, it sometimes treats the image as a "bag of patches." It sees the yogurt and it sees the strawberries, but doesn't work out how they relate.

This has a lot to do with being trained by a teacher who can't see the image. However good GPT-4's answers are, they rest only on text descriptions and bounding boxes; details that are in the image but not in the description are something the student never gets a chance to learn.

ScienceQA

ScienceQA is 21,000 multiple-choice science questions covering natural, social and language science; some questions come with an image, and some are text only. LLaVA writes its reasoning before its answer and, trained for 12 epochs, reaches 90.92% accuracy, close to the best result at the time, 91.68%.

The text-only GPT-4 with a two-example few-shot prompt reaches 82.69%, and many of the questions it fails are ones where it says the image or plot is missing. The authors tried two combinations:

  • GPT-4 as complement: when GPT-4 can't answer, use LLaVA's answer instead, 90.97%, about the same as LLaVA answering alone.
  • GPT-4 as judge: when the two give different answers, hand the question and both answers back to GPT-4 and ask it for a final answer, 92.53%, a new best result at the time.

A GPT-4 that can't see images helps even on questions that come with images. The authors' explanation is that some questions with images can actually be answered without looking at the image, and GPT-4 can recognize these and correct some of LLaVA's mistakes.

A few ablations on ScienceQA:

  1. Using CLIP features from the last layer gives 89.96%, 0.96 lower than the second-to-last layer. The authors suspect the last layer leans toward global, abstract image properties, while the second-to-last layer keeps more local detail.
  2. For the order of reasoning and answer, this comparison uses last-layer features. Answer first, trained for 12 epochs, gives 89.77%; reasoning first gets to 89.77% in just 6 epochs and barely improves after that. The authors conclude that reasoning first speeds up convergence but does little for the final result.
  3. Skipping the stage 1 feature alignment and training directly drops it to 85.81%.
  4. Switching the model from 13B to 7B drops it to 89.84%.

What came after

Half a year later, the same group of authors published LLaVA-1.5. The changes were few: the image resolution went up to 336 pixels, the linear projection was replaced with an MLP, and academic visual question answering data was added, paired with simple response-format prompts. The 13B version used only 1.2 million examples of public data, trained in about a day on a single machine with 8 A100s, and achieved the best results at the time on 11 benchmarks.

The "vision encoder + projection layer + language model" skeleton is easy to pick up, and many later open-source multimodal models kept it; scoring with GPT-4 as the judge also became more and more common in later model evaluations.

Conclusion

Instruction data doesn't have to be written by people, not even multimodal instruction data. LLaVA turned images into text and handed them to GPT-4 to write questions, getting 158K examples, then connected CLIP to Vicuna with a single linear projection. The two training stages together take a little over half a day on 8 A100s, yet its ability to chat about images already holds up against the examples in the GPT-4 report.

The training order in this paper is well worth copying. First freeze both ends and train only the projection layer, so the image features line up with the language model's word embeddings; then unfreeze the language model for instruction tuning. Skip the first step, and ScienceQA loses 5 percentage points.

So is there strawberry-flavored yogurt in the fridge or not? LLaVA says yes, with full confidence. Probably all its teacher ever told it was "There's yogurt in the fridge, and there are strawberries."