Scaling Instruction-Finetuned Language Models
- Publication
- Journal of Machine Learning Research 2024Journal
- First submitted to arXiv
- arXiv:2210.11416
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modeltuningflant52022,
title = {{Scaling Instruction-Finetuned Language Models}},
author = {Hyung Won Chung and Le Hou and Shayne Longpre and Barret Zoph and Yi Tay and William Fedus and Yunxuan Li and Xuezhi Wang and Mostafa Dehghani and Siddhartha Brahma and Albert Webson and Shixiang Shane Gu and Zhuyun Dai and Mirac Suzgun and Xinyun Chen and Aakanksha Chowdhery and Alex Castro-Ros and Marie Pellat and Kevin Robinson and Dasha Valter and Sharan Narang and Gaurav Mishra and Adams Yu and Vincent Zhao and Yanping Huang and Andrew Dai and Hongkun Yu and Slav Petrov and Ed H. Chi and Jeff Dean and Jacob Devlin and Adam Roberts and Denny Zhou and Quoc V. Le and Jason Wei},
year = {2022},
eprint = {2210.11416},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2210.11416}
}
In 2021, FLAN found that if you rewrite dozens of NLP datasets as instructions and use them to fine-tune a language model, the model gets better at tasks it has never seen. Around the same time, T0 and InstructGPT each fine-tuned models on instruction data too, with good results.
The next question follows naturally. Would more tasks and a bigger model do even better?
In this paper, Chung and colleagues at Google push both as far as anyone could at the time. The number of tasks grows from FLAN's 62 to 1,836, the largest model is the 540-billion-parameter PaLM, and chain-of-thought (CoT) data is added on top. The resulting model is called Flan-PaLM; the same pipeline was also used to fine-tune T5, from 80 million to 11 billion parameters, with the weights released, and that is the Flan-T5 so many people went on to use.
Defining the problem
Earlier instruction tuning studies each scaled up in only one direction. FLAN went up to a 137B model but had only 62 tasks; Natural Instructions v2 collected more than 1,600 tasks, but its models topped out at 11B. Nobody had tried scaling both at once.
The other problem is trickier. Almost all earlier instruction data used a "just give the answer" format, while one of PaLM's most celebrated abilities that year was chain-of-thought, writing out the reasoning steps before answering. Would fine-tuning on a pile of give-the-answer problems make the model forget how to reason step by step?
So this paper sets out to answer three things:
- How do results change when the number of tasks and the model size are scaled up together?
- Does instruction tuning hurt chain-of-thought reasoning? If so, how do you make up for it?
- Does the recipe still work on other architectures and other sizes?
Solving the problem
1,836 tasks

First, let's sort out the three units in the figure. A dataset is a raw data source, such as SQuAD. A task category is a type of problem, and the same SQuAD can be used for extractive question answering, question generation, context generation and more. A task is one combination of "dataset × task category," such as "generate questions from SQuAD."
Each of the four groups has its own source:
- Muffin (80 tasks): mostly FLAN's original tasks, plus new ones added in this paper, such as dialogue and program synthesis.
- T0-SF (193 tasks): the tasks from T0 that don't overlap with Muffin. SF stands for "sans Flan," that is, without Flan.
- NIV2 (1,554 tasks): Natural Instructions v2. The authors first removed 44 tasks related to MMLU, since MMLU is used for evaluation.
- CoT (9 tasks): nine datasets with human-written reasoning, such as GSM8K for arithmetic reasoning and StrategyQA for multi-step reasoning. The authors wrote about ten instruction templates for each dataset.
Altogether, that is 473 datasets, 146 categories and 1,836 tasks.
Each problem also shows up in several formats, in every combination of with or without exemplars and with or without written reasoning. Here is the paper's example:
[no exemplars, no reasoning]
Answer the following yes/no question. Can you write a whole Haiku in a single tweet?
→ yes
[with exemplars, with reasoning]
Q: Answer the following yes/no question by reasoning step-by-step. Could a dandelion suffer from hepatitis?
A: Hepatitis only affects organisms with livers. Dandelions don't have a liver. The answer is no.
Q: Answer the following yes/no question by reasoning step-by-step. Can you write a whole Haiku in a single tweet?
A: A haiku is a japanese three-line poem. That is short enough to fit in 280 characters. The answer is yes.
Only those nine CoT datasets use the "with reasoning" format.
The fine-tuning itself is cheap
Every model goes through the same pipeline, with only the learning rate, batch size, dropout and number of steps tuned. The optimizer is Adafactor with a constant learning rate and no decay; several short examples are packed into one sequence, with masking to block attention between examples.
Compared with pretraining, instruction tuning uses a pitifully small amount of compute. Fine-tuning Flan-PaLM 540B took only 0.2% of the pretraining compute, about 512 TPU v4 chips running for 37 hours; for the Flan-T5 sizes it ranges from 0.2% to 1.6%.
How it's tested
Evaluation uses only problems that don't appear in the fine-tuning data:
- MMLU: exam questions across 57 subjects, including mathematics, history, law and medicine, 5-shot.
- BBH: 23 hard tasks from BIG-Bench on which PaLM still couldn't beat the average human rater, 3-shot.
- TyDiQA: question answering in 8 languages, 1-shot.
- MGSM: GSM8K math word problems manually translated into 10 languages, 8-shot.
The authors deliberately skip the set of benchmarks used in the GPT-3 paper, because the training sets for nearly all of those tasks are already in the fine-tuning data.
MMLU and BBH are each tested twice, once answering directly and once writing a chain of thought before answering. The six scores are then combined into a single "normalized average." Normalizing means treating the random-guess score as zero and stretching the remaining range back to 0 to 100. For a two-choice question, for example, random guessing already gets 50%, so 55% correct counts as , and 45% correct counts as −10%.
Discussion
More tasks help, but the first 282 help most

The left panel looks at model size. The gray line is no fine-tuning, and the other lines add different numbers of tasks; at all three sizes, instruction tuning raises the normalized average by 9.4 to 15.5 percentage points. Each roughly tenfold increase in model size brings a clear rise in score, with or without fine-tuning.
The right panel looks at the number of tasks. Tasks are added group by group in the order CoT, Muffin, T0-SF, NIV2. The curve climbs fastest going from 0 to 282 tasks; adding more, up to 1,836, gains only a little.
The authors offer two explanations. One is that the tasks added later aren't diverse enough to teach the model anything new. The other is that most of what instruction tuning gains comes from teaching the model to "say out loud what it already learned in pretraining," and that kind of learning saturates past a certain point. The second explanation fits the numbers. PaLM read 780 billion tokens in pretraining, and instruction tuning used only 1.4 billion, about 0.2%.
The authors also leave open whether small or large models benefit more. In absolute score, 8B gains 15.5 and 540B gains 9.4; in relative reduction of the error rate, 540B gets 18.4% and 8B 16.6%. Same numbers, different calculation, and the conclusion flips.
Skip chain-of-thought practice, and it forgets how to reason

This is the figure most worth your time in the whole paper; look at the green line on the left. Fine-tuned only on give-the-answer data, the model does worse on evaluations that need chain-of-thought than with no fine-tuning at all; 540B drops from around 57 to around 42.
This seems to contradict earlier work, since FLAN and T0 both reported that instruction tuning helps on unseen tasks. The authors' explanation is that earlier evaluations only looked at "unseen NLP tasks," for example leaving out sentiment analysis in training and then testing on sentiment analysis; and earlier models were too small to do chain-of-thought well in the first place. In other words, what instruction tuning helps with is new tasks in the same prompting format as the fine-tuning data. Practice only direct answers during fine-tuning, and you lose out when tested on chain-of-thought.
The fix takes just nine CoT datasets. The blue line, fine-tuned on both kinds of data, is the highest of the four lines on the left; on the right, the blue line nearly overlaps the green line trained only on non-CoT data, so adding chain-of-thought data doesn't drag down the direct-answer questions.
These nine datasets have a side effect, too. Part of the CoT data is given without exemplars during fine-tuning, so Flan-PaLM learned zero-shot chain-of-thought. Add "Let's think step by step" after a question and its BBH score goes up, while the same phrase does nothing for PaLM without fine-tuning. Most of the successful examples in Kojima et al.'s earlier zero-shot chain-of-thought experiments used InstructGPT, and the authors speculate that this is because InstructGPT's fine-tuning data already contained something like chain-of-thought.
Headline results
Put the three together and you get Flan-PaLM 540B. On MMLU, chain-of-thought plus self-consistency (sampling several times and voting) reaches 75.2%, above PaLM's 69.3%, code-davinci-002's 68.3% and Chinchilla's 67.6%. On MGSM, the multilingual math problems, it scores 72.0%, with even Bengali at 69.6%; GSM8K is 83.9%, though GSM8K's training set is in the fine-tuning data.
There are places it didn't win, too. On the algorithmic tasks in BBH that test only symbol manipulation, such as tracking shuffled objects in order or sorting words alphabetically, Flan-PaLM still loses to code-davinci-002. On TyDiQA, it gains 14.9 percentage points over PaLM but still falls short of ByT5, which was fine-tuned directly on the TyDiQA training set.
Other models work too
The authors applied the same pipeline to models of different architectures, sizes and pretraining objectives. Here are a few rows:
| Model | Norm. avg. | MMLU direct | MMLU CoT | BBH direct | BBH CoT |
|---|---|---|---|---|---|
| T5-XL (3B) | −4.1 | 25.7 | 14.5 | 27.4 | 19.2 |
| Flan-T5-XL (3B) | 19.1 | 52.4 | 45.5 | 41.0 | 35.2 |
| T5-XXL (11B) | −2.9 | 25.9 | 18.7 | 29.5 | 19.3 |
| Flan-T5-XXL (11B) | 23.7 | 55.1 | 48.6 | 45.3 | 41.4 |
| PaLM 62B | 28.4 | 55.1 | 49.0 | 37.4 | 43.0 |
| PaLM 540B | 49.1 | 71.3 | 62.9 | 49.1 | 63.7 |
| Flan-PaLM 540B | 58.4 | 73.5 | 70.9 | 57.9 | 66.3 |
Compiled from Table 5 of the paper, showing selected rows and columns.
T5 benefits the most. Without fine-tuning, T5 (here the LM-adapted version, additionally trained with a language-modeling objective) is close to guessing on these hard evaluations; its MMLU chain-of-thought score is only 14.5%, lower than random guessing on four-choice questions, so its normalized average is negative. After fine-tuning, the 3B Flan-T5-XL scores 52.4% on MMLU, higher than GPT-3 175B's 43.9%; the 11B Flan-T5-XXL also beats PaLM 62B on BBH direct answering, 45.3 against 37.4 in the table.
The strongest model in the whole paper is Flan-U-PaLM, which is PaLM 540B first given continued pretraining with the UL2 objective and then instruction tuned. The authors conclude that the two methods stack, and neither one requires making the model bigger.
Nicer to use
Beyond the scores, the authors also had people compare open-ended answers from PaLM 540B and Flan-PaLM 540B. The evaluation set has 190 examples covering categories such as creativity, reasoning, planning and explanation, in both zero-shot and few-shot settings. Flan-PaLM was preferred 79% of the time; when the prompt added "let's think step by step," the preference margin widened by about another 10%.

In the first problem, , so and . PaLM doesn't answer; it treats the problem as the start of an exam paper and keeps writing question after question. On the second, it copies the question once and then keeps writing the same story, which has nothing to do with the question.
The authors saw three kinds of problems in PaLM's zero-shot output: continuing with related text instead of answering, repeating the question with small modifications, and not knowing when to stop. They suspect this comes from not using end-of-sequence tokens during pretraining. Instruction tuning teaches the model to stop once it has answered, which matters especially for ordinary users who aren't doing prompt engineering.
What came after
The paper's last sentence recommends instruction tuning for virtually all pretrained language models. Things since have largely followed that advice, and releasing a base version alongside an instruction-tuned version has become almost standard practice. LLaMA, four months later, also followed this paper's recipe to give its 65B model a simple round of instruction tuning, raising MMLU from 63.4% to 68.9%.
In early 2023, Google packaged the whole set of tasks and templates and released it as The Flan Collection. A few months later, though, the wind shifted. LIMA fine-tuned LLaMA 65B on just 1,000 carefully chosen question-answer pairs and got it writing quite good answers; people started caring more about the quality and origin of instruction data, such as questions from real users, rather than just the number of tasks.
Conclusion
This paper pushes all three directions of instruction tuning at once. Tasks went up to 1,836, with most of the gain concentrated in the first 282; the model grew tenfold, and the gain stayed clear; and chain-of-thought data from just nine datasets was enough to recover the reasoning lost to direct-answer training. The cost was only 0.2% to 1.6% of pretraining compute.
The most practical reminder is that the model gets good at whatever format the fine-tuning data takes. Fine-tuned only on direct-answer data, 540B's normalized average on evaluations that need chain-of-thought actually falls from about 57 without fine-tuning to about 42; adding nine chain-of-thought datasets is what avoids that loss.
For most people, what this paper left behind is that handful of Flan-T5 checkpoints. PaLM was never released, so however good Flan-PaLM's scores look, you can only admire them in the paper.