Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Publication
- ACL 2023Conference
- First submitted to arXiv
- arXiv:2212.10560
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modeltuningselfinstruct2022,
title = {{Self-Instruct: Aligning Language Models with Self-Generated Instructions}},
author = {Yizhong Wang and Yeganeh Kordi and Swaroop Mishra and Alisa Liu and Noah A. Smith and Daniel Khashabi and Hannaneh Hajishirzi},
year = {2022},
eprint = {2212.10560},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2212.10560}
}
Instruction tuning needs instruction data, and at the time, nearly all instruction data was written by people.
Academia's approach was to rewrite existing NLP datasets as instructions, as in FLAN and T0, or to have researchers jointly write more than 1,600 tasks, as in Super-NaturalInstructions (SuperNI from here on). On the industry side, InstructGPT used a large number of real user prompts and had labelers write demonstrations, and none of that data was ever released.
The former has two problems. The tasks people manage to write are mostly the classification, question answering and summarization that researchers know well, some distance from what ordinary people actually ask a model to do; and getting people to come up with new tasks and then write good answers takes both creativity and expertise, which is hard to do at scale.
In December 2022, Wang and colleagues at the University of Washington came up with a method that looks a little lazy. Since GPT-3 can write just about anything, why not let it write its own questions, answer them itself, and then fine-tune it on that data?
Defining the problem
First, let's pin down what instruction data looks like. A task has an instruction and one or more input-output pairs , and the model has to achieve .
The line between instruction and input is often blurry. "Write an essay about school safety" can be taken whole as the instruction, with no input needed; or "Write an essay about the following topic" can be the instruction, with "school safety" as the input. The authors allow the input to be empty, to make the data format more varied.
The question this paper wants to answer is whether a language model that has never been instruction tuned, plus a very small number of human-written seeds, can produce instruction data good enough to teach the model itself to follow instructions.
Solving the problem

The whole pipeline is a loop. The authors and their labmates first hand-wrote 175 seed tasks, each with one instruction and one input-output pair, and put them in the task pool; the next four steps are all handed to GPT-3 (davinci, the largest model on the OpenAI API), and new tasks that pass filtering go back into the pool to generate the next batch.
Step 1: write new questions
Each time, 8 instructions are drawn from the task pool as examples, 6 from the human-written seeds and 2 from tasks the model generated earlier, and the model is asked to continue with new instructions. Mixing in the model's own output is meant to make the questions more varied as the pool snowballs; keeping most of the examples human-written keeps the quality from drifting.
Step 2: is it a classification question?
Next, each instruction needs inputs and outputs, and classification questions are handled differently from the rest, so they have to be told apart first. The authors use few-shot prompting, showing the model 12 classification instructions and 19 non-classification instructions and letting it decide which kind a new instruction is. A classification question here means a task whose output is one of only a few labels.
Step 3: generate instances
Ordinary questions use input-first, writing the input from the instruction and then the output. This is the same order in which you'd normally use a model.
Doing the same for classification questions causes trouble. Take grammar error detection. The inputs the model writes are almost all grammatically correct sentences, so the labels are heavily skewed to one side. Classification questions therefore use output-first, listing the possible labels first and then writing an input for each label. For the question "determine whether this text is in favor of or against abortion rights," for example, the model first settles on the label "Pro-abortion" and then generates a passage in favor of abortion rights.
Step 4: filtering
The model repeats itself, so each new instruction is compared by ROUGE-L against every instruction in the task pool, and accepted only if its similarity to all of them is below 0.7. ROUGE-L looks at the longest common subsequence of two sentences. Take “Give me a quote from a famous person on this topic.” from the paper's pipeline figure: the candidate “Give me a famous quote on this topic.” merely rephrases it and gets blocked. “Find out if the given review is positive or negative.” reuses the pattern of the figure's other instruction, “Find out if the given text is in favor of or against abortion.”, but switches the task from judging a stance to judging whether a review is positive or negative, so it stays.
There are other filtering rules as well. Instructions containing keywords like image, picture or graph, which a language model can't handle, are dropped; so are identical instances under the same instruction, and instances with the same input but different outputs; and some heuristic rules remove generations that are too long, too short, or whose output just repeats the input.
Fine-tuning
Once the data is ready, it is used to fine-tune the same GPT-3 that generated it, through OpenAI's fine-tuning API, with no loss on the prompt and 2 epochs of training. To keep the model from being picky about format, instructions and inputs are joined with a variety of templates, for example with or without "Task:" before the instruction, with or without "Input:" before the input, and with different numbers of blank lines in between. The resulting model is called GPT3Self-Inst.
What it produced
The end result is 52,445 instructions and 82,439 instances, of which 11,584 instructions are classification questions and 35,878 instances have no input.
How far are these questions from the seeds? The authors computed the ROUGE-L between each new instruction and its most similar seed. Most fall between 0.2 and 0.45, peaking around 0.3, and very few exceed 0.6, meaning most are new questions not found among the seeds. The verbs and objects are also widely spread; the 20 most common verbs plus the 4 most common objects of each make up only 14% of all instructions.
Quality is another matter. The authors randomly sampled 200 instructions, one instance each, and checked them one by one themselves:
| Check | Passed |
|---|---|
| The instruction describes a valid task | 92% |
| The input is appropriate for the instruction | 79% |
| The output is a correct and acceptable response | 58% |
| All three are fine | 54% |
Compiled from Table 2 of the paper.
Nearly half the instances have errors. The authors' view is that even when the content is wrong, the format is mostly right, or at least partly right, which still helps teach the model "how to respond to an instruction."
Discussion
SuperNI: nearly catching up with InstructGPT001
The first set of experiments uses SuperNI's 119 evaluation tasks, 100 instances each, answered zero-shot with only the task definition given, and measures the ROUGE-L between the output and the reference answer:
| Model | Parameters | ROUGE-L |
|---|---|---|
| GPT-3 | 175B | 6.8 |
| T0 | 11B | 33.1 |
| GPT-3 + T0 training data | 175B | 37.9 |
| GPT3Self-Inst | 175B | 39.9 |
| InstructGPT001 | 175B | 40.8 |
| Tk-Instruct (trained on SuperNI) | 11B | 46.0 |
| GPT-3 + SuperNI training data | 175B | 49.5 |
| GPT3Self-Inst + SuperNI training data | 175B | 51.6 |
Compiled from Table 3 of the paper.
The original GPT-3 barely follows instructions; it often writes irrelevant or repetitive content and doesn't know when to stop. After fine-tuning on its own generated data, ROUGE-L rises by 33.1, less than a point behind InstructGPT001, which was trained on private user data and human labels. Only the 001 version is compared here, because newer versions were trained on the latest user data and quite possibly saw SuperNI's test set.
Training on SuperNI's own training data is of course stronger, since the question style and format are identical. But adding Self-Instruct data on top of the SuperNI training data still lifts the score from 49.5 to 51.6, which shows the two kinds of data complement each other.
Real uses: 252 new questions
SuperNI's tasks are mostly designed for research and lean toward classification. The authors separately brainstormed some domains where language models might come in handy, such as writing email, social media, productivity tools, entertainment and programming, and wrote instructions for each domain, 252 questions in total.
These questions are hard to score automatically. Ones like writing code or turning first-order logic into natural language are beyond what ordinary crowdworkers can rate. So the authors who wrote the questions rated them themselves, on four levels. A: correct and satisfying; B: acceptable, with minor flaws; C: responds to the instruction, but with significant errors; D: irrelevant or invalid.

Left of the dashed line are GPT-3 and its various fine-tuned versions; to the right are three generations of OpenAI's InstructGPT. Every model answers the same 252 questions: 44.4% of GPT3Self-Inst's answers are rated A, against 50.8% for InstructGPT001.
Counting B as usable too, GPT3Self-Inst is at 63.9% and InstructGPT001 at 68.7%, less than 5 percentage points apart, which is the "only a 5% absolute gap" in the abstract. It also clearly beats GPT-3 fine-tuned on T0 or SuperNI data. InstructGPT002 and 003, however, are in another league; more than three quarters of 003's responses were rated A.
More data helps, but quality matters more

The blue line comes from fine-tuning on different numbers of Self-Instruct instructions. The proportion rated A rises from 31.0% to 44.4%, but barely moves after about 16,000. On SuperNI it saturates even earlier, topping out at a few hundred instructions; the authors suspect this is because this data is very different from typical NLP tasks.
The orange point is more interesting. The authors kept all the instructions and inputs, had InstructGPT003 rewrite only the outputs, and then used that to fine-tune GPT-3. The proportion rated A jumps straight to 54.4%, 10 percentage points higher. In other words, InstructGPT003's capability is distilled onto this set of questions.
The cost of this experiment is also listed in the appendix. At the time, davinci charged $0.02 per 1,000 tokens, and generating the whole dataset cost about $600; fine-tuning GPT-3 on all the data cost $338.
Limitations
The authors list three themselves:
- The long tail: models are good at what is common in their pretraining data, so Self-Instruct's gains may also be concentrated on common tasks, and it is likely to break down on rare or highly creative instructions.
- Dependence on large models: the method relies on the language model's own inductive biases, may work only on large models, and is out of reach for people without enough compute.
- Amplifying bias: repeated self-generation may keep amplifying social biases; the difficulty of balancing labels for classification questions also reflects the model's existing preferences.
One more point is worth adding. Self-Instruct brings the model no new knowledge, since both questions and answers come from GPT-3 itself; what it teaches is "how to respond when you see an instruction." Just having InstructGPT003 rewrite the outputs lifts the A rate from 44.4% to 54.4%, which bears this out too.
What came after
Three months later, Stanford's Alpaca carried this pipeline over to LLaMA. Starting from the same 175 seeds, they switched to text-davinci-003 to generate 52,000 instruction demonstrations; generating the data cost less than $500, and fine-tuning LLaMA-7B less than $100. Alpaca's announcement is also clear that it is restricted to academic research, partly because LLaMA itself has a non-commercial license, and partly because OpenAI's terms of use prohibit using its outputs to develop models that compete with OpenAI.
After that, generating instruction data with a strong model became common practice, and the methods grew more and more elaborate. In 2024, Tülu 3 first prepared a large number of "personas," then had GPT-4o play these people and write questions, to increase the diversity of the questions; this is a direct descendant of Self-Instruct's idea of mixing in model-generated tasks.
Conclusion
With only 175 human-written seed tasks, Self-Instruct had GPT-3 produce more than 50,000 instructions. After fine-tuning, it caught up with InstructGPT001 on SuperNI and trailed by less than 5 percentage points on 252 real-use questions. Generating the data cost about $600, and fine-tuning GPT-3 on all of it cost $338.
If you want to build your own instruction data, the paper's four steps still look very practical today. Mix in a few model-generated tasks for diversity, settle the label before writing the input for classification questions, block duplicates by similarity, and clear out obvious junk with rules. Quality is what to watch most closely. Of the 200 the authors spot-checked, only 54% had instruction, input and output all in order. Usable, but don't trust it too much.
Most of the people who followed kept only the question-writing pipeline and swapped the "self" for a stronger teacher. Alpaca had text-davinci-003 write the questions and answers, and Tülu 3 had GPT-4o play all kinds of personas to write questions; in the paper, just having InstructGPT003 rewrite the outputs added 10 percentage points to the A rate, which had already hinted at this step.