Finetuned Language Models Are Zero-Shot Learners
- Publication
- ICLR 2022Conference
- First submitted to arXiv
- arXiv:2109.01652
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modeltuningflan2021,
title = {{Finetuned Language Models Are Zero-Shot Learners}},
author = {Jason Wei and Maarten Bosma and Vincent Y. Zhao and Kelvin Guu and Adams Wei Yu and Brian Lester and Nan Du and Andrew M. Dai and Quoc V. Le},
year = {2021},
eprint = {2109.01652},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2109.01652}
}
GPT-3 showed everyone what few-shot learning looks like. Put a few examples in the prompt, and the model follows along.
Take the examples away, and things get much worse. On tasks like reading comprehension, question answering and natural language inference (NLI), GPT-3's zero-shot scores trail its few-shot scores by a clear margin. The authors' guess is that without examples, the prompts the model faces look too different from the format of its pretraining data. NLI, for example, asks whether one sentence can be inferred from another, and questions like that almost never show up naturally in web text.
In September 2021, Wei, Bosma, Zhao, Guu and colleagues at Google proposed a very direct approach. Many NLP tasks can be described in a single sentence, such as “Is this movie review positive or negative?”, so why not rewrite dozens of existing datasets as natural language instructions, fine-tune a language model on them, and see whether it can handle tasks it has never seen? They called this instruction tuning, and the fine-tuned model FLAN (Finetuned Language Net). The name later became more famous than the model itself; Flan-T5 and Flan-PaLM both carried it on.
Defining the problem

The top half of the figure is the design of the whole experiment. During training, the model sees many tasks, such as commonsense reasoning, translation and sentiment analysis, each written as an instruction; at test time, it gets a natural language inference question, even though during training it never saw a single inference task.
The bottom half previews the results. On three unseen task types, FLAN 137B's zero-shot performance beats GPT-3 175B's zero-shot, and even beats GPT-3's few-shot.
The authors place instruction tuning between two existing approaches. One is the “pretrain, then fine-tune” approach of BERT and T5, where each task needs plenty of labeled data and you end up with a specialist model. The other is GPT-3's “prompting,” with no fine-tuning, relying on few-shot examples or carefully engineered prompts. Instruction tuning fine-tunes on supervision from many tasks, but the goal is for the model, at inference time, to understand a natural language description of a task it has never seen.
What most needs pinning down here is the definition of “unseen.” Requiring only that the test dataset not appear in training isn't enough. If the model trained on MNLI and is then tested on RTE, both are NLI, and it's hard to claim it's doing a new task.
Solving the problem
Rewriting datasets as instructions
The authors gathered 62 public text datasets from TensorFlow Datasets, covering both understanding and generation tasks, and grouped them by task type into 12 clusters, such as natural language inference with 7 datasets, reading comprehension with 5, closed-book QA with 3, translation with 8 and summarization with 11.
For each dataset, 10 different instruction templates were written by hand. Most templates describe the original task, but for more diversity, up to 3 templates per dataset “turn the task around”; a sentiment classification dataset, for instance, is also used to ask the model to write a movie review. The same NLI example can be cast into several phrasings:
Template 1:
<premise>
Based on the paragraph above, can we conclude that <hypothesis>?
<options>
Template 3:
Read the following and determine if the hypothesis can be inferred from the premise:
Premise: <premise>
Hypothesis: <hypothesis>
<options>
During training, each example is formatted with a template picked at random from its dataset's templates.
Options for classification tasks
FLAN is a decoder-only language model, so it can answer generation tasks directly. Classification tasks are trickier. GPT-3 used “rank classification,” for example comparing only the probabilities of the two words “yes” and “no” and taking the higher one. The authors point out that there may be many ways of saying “yes,” so the probability gets spread out.
So for classification tasks, FLAN appends OPTIONS: and all the classes to the end, letting the model know which answers to choose from, like the “yes / it is not possible to tell / no” in the NLI question in the figure.
Holding out entire task types
For evaluation, the authors adopt a fairly conservative definition. A dataset counts as an unseen task only if its entire cluster was left out of fine-tuning. To test NLI, all 7 datasets in the NLI cluster are removed and the model is fine-tuned on the remaining clusters; closely related clusters are excluded too, so testing NLI also means leaving out the “paraphrase detection” cluster.
This means evaluating clusters requires training different FLAN models, each with one cluster held out. It costs more, but what gets tested really is “an unseen task type.”
Training setup
The base model is LaMDA-PT, a 137-billion-parameter decoder-only language model. It was pretrained on web pages (including code), dialog data and Wikipedia, 2.49 trillion BPE tokens in total, about 10% of them non-English. LaMDA-PT has had only language model pretraining, without the dialog fine-tuning that later went into LaMDA.
To balance dataset sizes, each dataset contributes at most 30,000 examples, mixed with T5's “examples-proportional mixing” (the very scheme the T5 paper compared against the alternatives) and a mixing rate maximum of 3,000. Say three datasets have 100,000, 20,000 and 1,000 examples. After the 30,000 cap they're at 30,000, 20,000 and 1,000; the mixing rate maximum counts each of the first two as just 3,000, so the sampling ratio is 3:3:1. Large datasets don't drown out the small ones.
Fine-tuning runs for 30,000 steps with 8,192 tokens per batch, using Adafactor at a learning rate of , with input and output lengths of 1,024 and 256. It takes about 60 hours on a 128-core TPUv3, and the number of gradient steps is under 2% of pretraining.
Discussion
Main results
FLAN is compared against three baselines. The most direct is the same LaMDA-PT without fine-tuning, run zero-shot and few-shot with the prompts from the GPT-3 paper; this comparison directly measures how much instruction tuning itself adds. The other two are GPT-3 175B and GLaM 64B/64E, with numbers taken from their respective papers.
FLAN's scores are computed two ways. One is the average over all templates, roughly what you'd get by writing any natural instruction; the other picks the best template on the dev set and reports its test score, matching the case in the GPT-3 paper where a dev set is available for tuning the prompt. With the best template, FLAN's zero-shot beats GPT-3's zero-shot on 20 of 25 datasets, and on 10 of them it even beats GPT-3's few-shot.
Here are a few of the datasets:
| Dataset | LaMDA-PT zero-shot | GPT-3 zero-shot | GPT-3 few-shot | FLAN zero-shot (template average) | FLAN zero-shot (best template) |
|---|---|---|---|---|---|
| ANLI R1 | 39.6 | 34.6 | 36.8 | 47.7 ± 1.4 | 46.4 |
| RTE | 73.3 | 63.5 | 72.9 | 78.3 ± 7.9 | 84.1 |
| CB | 42.9 | 46.4 | 82.1 | 64.1 ± 14.7 | 83.9 |
| BoolQ | 81.0 | 60.5 | 77.5 | 80.2 ± 3.1 | 82.9 |
| OBQA | 41.8 | 57.6 | 65.4 | 77.4 ± 1.3 | 78.4 |
| ARC-c | 42.0 | 51.4 | 51.5 | 61.7 ± 1.4 | 63.1 |
| HellaSwag | 57.0 | 78.9 | 79.3 | 56.4 ± 0.5 | 56.7 |
| Winogrande | 68.3 | 70.2 | 77.7 | 67.3 ± 2.5 | 71.2 |
Compiled from Table 2 in the paper's appendix; values are accuracy.
NLI improves the most. The authors cite the GPT-3 paper's observation that NLI questions rarely appear naturally in web text and are awkward to write as sentence completions; FLAN, by contrast, can simply ask, “Does mean that ?” Closed-book QA (ARC) and reading comprehension (MultiRC, OBQA) also improve substantially.
The table also shows how sensitive the model is to the instructions. CB's template average is 64.1, with a standard deviation as high as 14.7, so rephrasing the question can move the score by more than ten points; the best template reaches 83.9.
The last two rows are counterexamples. HellaSwag and Winogrande are “finish the sentence” tasks to begin with, almost identical to the language model's pretraining objective, so the instructions are redundant. On commonsense reasoning and coreference resolution tasks of this kind, FLAN beats the untuned LaMDA-PT on only 3 of 7. On HellaSwag, GPT-3's 78.9 is far above LaMDA-PT; that gap comes from the base model, and instruction tuning did nothing to close it.
Translation falls somewhere in between. FLAN beats GPT-3's zero-shot in all 6 translation directions but loses to GPT-3's few-shot in most of them. It does well translating into English and is weaker translating out of English, which the authors attribute to the English-dominated SentencePiece vocabulary and pretraining data.
Ablations: number of clusters, model size, the instructions themselves
More clusters, better performance. The authors hold out three fixed clusters, NLI, closed-book QA and commonsense reasoning, and add the other clusters for fine-tuning one at a time, going from 1 to 7 (11 to 39 datasets). Average performance on the three held-out clusters climbs from 49.9 all the way to 63.5; the one exception is the step that adds the sentiment analysis cluster, which barely helps. Even at 7 clusters the curve hasn't saturated.

Small models actually get worse. Using the same cluster split, the authors instruction-tune five models ranging from 422 million to 137 billion parameters. The black line is the untuned model and the blue line is after tuning. 68B and 137B improve a lot, with 137B going from about 54 to about 68; but at 8B and below, scores after instruction tuning are actually lower than without it.
The authors offer a hypothesis. For a small model, learning the roughly 40 fine-tuning tasks uses up the model's capacity, leaving nothing over for generalizing to new tasks; a large model still has capacity to spare after learning these tasks, and picks up “following instructions” along the way. The paper doesn't test this explanation further.

The instructions themselves are key. Could the improvement just be an effect of multi-task fine-tuning, with the instructions playing no role? The authors tried two kinds of fine-tuning without instructions. One gives only inputs and outputs; for translation, the input is “The dog runs.” and the output “Le chien court.” The other prepends the task and dataset name to the input, as in “[Translation: WMT'14 to French] The dog runs.”
The inputs-and-outputs-only version, evaluated with instructions, averages just 37.3. The version with dataset names scores 46.6 when evaluated with instructions and 47.0 when evaluated with names. FLAN, fine-tuned and evaluated with instructions, scores 55.2. Multi-task data does help, but to generalize to unseen tasks, the tasks have to be described in natural language during training.
Combining with other methods
Instruction tuning also works together with few-shot examples. At inference time, the authors put up to 16 examples into the same instruction format and place them before the question. Scores improve further on every cluster, most of all on tasks with more complex output formats such as translation, struct-to-text and closed-book QA; the standard deviation across templates also shrinks, so the model is less sensitive to how the prompt is worded.
Another experiment is prompt tuning, which freezes the model and trains only a short sequence of continuous vectors placed in front of the input. On SuperGLUE, prompt tuning starting from FLAN beats starting from LaMDA-PT in every case; the authors mention that, especially with only 32 training examples, the improvement often exceeds 10%.
Limitations
The authors list several themselves. Assigning tasks to clusters involves some subjectivity. The instructions are all short, usually a single sentence, very different from the detailed instructions given to crowd workers. Test examples may appear in the pretraining web data, though the post-hoc analysis in the appendix found that data overlap had no noticeable effect on the results. And a 137-billion-parameter model is expensive to deploy.
How later work responded
Around the same time, BigScience, led by Hugging Face, released T0, which used a similar approach for prompted multi-task training on an 11-billion-parameter T5 and also saw gains in zero-shot ability. That doesn't fully agree with FLAN's finding that models at 8B and below get worse; the two differ in base model and in the number of tasks and templates.
A year later, Chung et al. at Google expanded the fine-tuning tasks to 1,836 in Scaling Instruction-Finetuned Language Models and added chain-of-thought data. On benchmarks such as MMLU and BBH, even the 80-million-parameter Flan-T5-Small improves over the original T5-Small. Those benchmarks differ from this paper's “hold out an entire task type” setup, so the two conclusions can't be compared directly; what the result shows is that whether a small model benefits from instruction tuning depends heavily on the number and diversity of tasks.
OpenAI tested it from another angle in the InstructGPT paper. They fine-tuned the 175B GPT-3 on the FLAN and T0 data, then compared the results on real API user prompts. Both models beat the original GPT-3 but lost to a model fine-tuned with supervision only on labeler demonstrations; InstructGPT's outputs were preferred 78% of the time over the FLAN version and 79% over the T0 version. Their explanation is that public NLP datasets are mostly classification and QA tasks that are easy to score automatically, which make up only about 18% of real usage, against about 57% for open-ended generation and brainstorming.
Conclusion
Rewrite 62 existing datasets as instructions, fine-tune a 137B model on them, and it beats GPT-3's zero-shot on task types it has never practiced. The real effort went into the experimental design, which held out entire task types as the exam and went out of its way to show that the gains come from the instructions themselves.
If you're preparing instruction data yourself, there are two things you can copy directly. The more kinds of tasks, the better; and it's best to prepare several phrasings for each task, since rewording alone can swing the score by more than ten points.
These days, instruction tuning makes people think of chatbots. It started out as a stack of rewritten NLP exam papers.