Back to paper library
arXiv 2022NeurIPS 2022Model adaptationPaper notes

InstructGPT

What do you actually want?

Original paper

Training language models to follow instructions with human feedback

Paper authors

Long Ouyang · Jeff Wu · Xu Jiang …

Show all 20 authors

Long Ouyang · Jeff Wu · Xu Jiang · Diogo Almeida · Carroll L. Wainwright · Pamela Mishkin · Chong Zhang · Sandhini Agarwal · Katarina Slama · Alex Ray · John Schulman · Jacob Hilton · Fraser Kelton · Luke Miller · Maddie Simens · Amanda Askell · Peter Welinder · Paul Christiano · Jan Leike · Ryan Lowe

Publication
NeurIPS 2022Conference
First submitted to arXiv
arXiv:2203.02155
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modeltuninginstructgpt2022,
  title = {{Training language models to follow instructions with human feedback}},
  author = {Long Ouyang and Jeff Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray and John Schulman and Jacob Hilton and Fraser Kelton and Luke Miller and Maddie Simens and Amanda Askell and Peter Welinder and Paul Christiano and Jan Leike and Ryan Lowe},
  year = {2022},
  eprint = {2203.02155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.02155}
}
On this page

OpenAI's January 2022 blog post includes an example. Feed GPT-3 "Explain the moon landing to a 6 year old in a few sentences," and instead of answering, it carries on with "Explain the theory of gravity to a 6 year old," "Explain the theory of relativity," "Explain the big bang theory."

GPT-3's training objective is to predict the next word on a web page. The web is full of lists of questions, so following one question with the next is a perfectly reasonable continuation. But what the user wants is an answer. The OpenAI authors describe this gap as the model being "misaligned": its language-modeling objective is not the same as "follow the user's instructions helpfully and safely."

In this paper, Ouyang, Wu, Jiang, Lowe, Leike, Christiano and others on OpenAI's alignment team take the human feedback pipeline validated on summarization over the previous two years and apply it to real user prompts sent to the OpenAI API. The most quoted result is that labelers prefer the 1.3-billion-parameter InstructGPT to the 175-billion-parameter GPT-3, a model with more than a hundred times as many parameters.

Nine months later, OpenAI launched ChatGPT, and its official announcement called it a "sibling model" to InstructGPT, trained with the same RLHF methods but with slight differences in the data collection setup.

Defining the problem

The authors borrow a framing proposed by Askell et al. at Anthropic and split "alignment" into three parts:

  • Helpful: help the user accomplish what they are trying to do.
  • Honest: don't make up information or mislead the user.
  • Harmless: don't cause physical, psychological or social harm to people or the environment.

All three are hard to measure directly. The authors' main metric is labeler preference, that is, which of two models' responses to the same prompt is better. On honesty, the authors admit they cannot measure the model's "beliefs," so they fall back to measuring "truthfulness," meaning whether what the model says matches the facts. They measure it in two ways: how much information the model makes up that isn't in the source on closed-domain tasks (such as summarization), and the TruthfulQA benchmark. On harmlessness, they first asked labelers to judge whether an output was "potentially harmful," then found that this required guessing how the output would be used, so they switched to a few more concrete proxy criteria, such as whether an output is inappropriate in a customer-assistant context, denigrates a protected group, or contains sexual or violent content.

So the question this paper sets out to answer is how much human feedback can improve GPT-3 on the wide range of instructions users actually send, and at what cost.

Solving the problem

The three steps

Original Figure 2. The three steps of the method. Step one samples a prompt from the prompt dataset (explain the moon landing to a 6 year old), a labeler writes a demonstration answer, and that is used to fine-tune GPT-3 with supervised learning. Step two samples several model outputs (A to D) for the same prompt, a labeler ranks them from best to worst (D > C > A = B), and the rankings are used to train the reward model. Step three samples a new prompt (write a story about frogs), the policy generates an output, the reward model computes a reward r_k, and PPO uses it to update the policy.
Original Figure 2, Ouyang et al., from PDF page 3, arXiv license, scholarly quotation.

The pipeline follows Ziegler et al. (2019) and Stiennon et al. (2020), with an extra demonstration step at the front:

  1. Collect demonstrations and do supervised fine-tuning (SFT): labelers write ideal responses to prompts, which are used to fine-tune GPT-3.
  2. Collect comparisons and train a reward model (RM): sample several model outputs for the same prompt, have labelers rank them, and train a reward model that predicts human preferences.
  3. Optimize the policy with PPO: use the reward model's score as the reward and keep fine-tuning the SFT model with reinforcement learning.

Steps 2 and 3 can be repeated, sampling from the current best policy to collect new comparison data. The authors say that in practice most of the comparison data comes from SFT model outputs, with a smaller share from PPO models.

Where the data comes from

The prompts come mainly from the OpenAI API Playground. Users there were working with an earlier InstructGPT version that had only been through supervised fine-tuning, and the interface kept reminding them that their data might be used to train new models; API data from production use was not used. The authors deduplicated prompts by common prefix, took at most 200 prompts per user, and split training, validation and test sets by user ID, so that test prompts come from users never seen during training. Personally identifiable information was also filtered out of the training set.

At the very start, there were still few instruction-style prompts on the API, so the labelers wrote the first batch of prompts themselves. These came in three kinds: arbitrary tasks, a single instruction plus several question-answer examples, and prompts based on the use cases that applicants on the API waitlist had described.

The three datasets ended up at these sizes:

DatasetPurposeTraining prompts
SFTLabeler demonstrationsAbout 13,000 (11,295 labeler-written, 1,430 from the API)
RMLabeler rankingsAbout 33,000 (6,623 labeler-written, 26,584 from the API)
PPOPrompts only, no labels31,144 (all from the API)

Compiled from Table 6 in the paper's appendix.

The mix of use cases in these prompts looks very different from NLP datasets. By the labelers' classification, generation makes up 45.6%, open QA 12.4%, brainstorming 11.2%, chat 8.4% and rewriting 6.6%; summarization, classification, closed QA and extraction add up to only a little over 10%. Over 96% of the data is in English.

There were about 40 labelers, recruited through Upwork and Scale AI. They first had to pass a screening test that checked whether they could spot sensitive prompts and judge whether something was harmful, and how closely their judgments matched the researchers'. During training, the priority was helpfulness to the user; for the final evaluations, labelers were asked to put truthfulness and harmlessness first. A separate group of "held-out" labelers, who contributed no training data and took no screening test, was brought in to check whether the model had only learned to please one particular group of people.

Reward model: ranking several at once

The SFT model is trained on the labeler demonstrations for 16 epochs. The authors found that it already starts to overfit on validation loss after 1 epoch, but training for more epochs actually improves both the reward model score and human preference, so in the end they select the SFT model by reward model score.

The reward model starts from the SFT model, with the final output layer removed and replaced by one that outputs a scalar. To speed up collection, labelers see 4 to 9 responses at a time and rank them directly. KK responses yield (K2)\binom{K}{2} pairwise comparisons, and the loss is the same as in Stiennon et al., except that it is averaged over all the pairs:

loss(θ)=−1(K2)E(x,yw,yl)∼D[log⁡σ(rθ(x,yw)−rθ(x,yl))]\text{loss}(\theta) = -\frac{1}{\binom{K}{2}}\mathbb{E}_{(x, y_w, y_l)\sim D}\left[\log\sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)\right]

ywy_w is the better response in a pair and yly_l the worse one. Let's work it through with the ranking D > C > A = B from Figure 2:

python
import itertools
import torch
import torch.nn.functional as F

# The reward model runs one forward pass for each of the four responses to the same prompt
r = {"A": torch.tensor(0.1), "B": torch.tensor(0.3),
     "C": torch.tensor(0.9), "D": torch.tensor(1.5)}
rank = {"D": 1, "C": 2, "A": 3, "B": 3}   # labeler ranking: D > C > A = B

pairs = [(w, l) if rank[w] < rank[l] else (l, w)
         for w, l in itertools.combinations(r, 2) if rank[w] != rank[l]]
print(pairs)   # [('C', 'A'), ('D', 'A'), ('C', 'B'), ('D', 'B'), ('D', 'C')]

loss = torch.stack([-F.logsigmoid(r[w] - r[l]) for w, l in pairs]).mean()
print(round(loss.item(), 4))   # 0.346

4 responses give 6 pairs. A and B are tied, so that pair is dropped, leaving 5.

The authors call out one implementation pitfall in particular. The pairs drawn from a single ranking are highly correlated with one another. If they are shuffled into the whole dataset, each response gets used K−1K-1 times within one epoch, and the reward model overfits quickly. Instead, they put all the pairs for one prompt into a single batch element. Each response then needs only one forward pass, cutting the computation from (K2)\binom{K}{2} passes to KK, and validation accuracy and log loss also improve noticeably.

The reward model is always 6B, because a 175B reward model trained unstably and was also unsuitable for initializing the PPO value function; the same 6B reward model is used for policies of every size. Before moving on to RL, the authors shift the scores so that labeler demonstrations have a mean score of 0.

PPO plus pretraining gradients

The RL stage is set up as a bandit environment. It presents a random user prompt, the policy answers once, the reward model scores the answer, and the episode ends. As in Stiennon et al., a per-token KL penalty against the SFT model is added to keep the policy from wandering too far just to please the reward model, and the value function is initialized from the reward model. This version is called PPO.

The authors found that PPO alone makes the model regress on some public NLP benchmarks, such as SQuAD, DROP, HellaSwag and WMT 2015 French-to-English translation. They call this the "alignment tax": paying for alignment with lost capability on some tasks. If the alignment tax is too high, people will tend to use models that are more capable but unaligned, so they want to keep that tax low.

The fix is to mix language-modeling gradients from the pretraining data in with the PPO gradients:

objective(ϕ)=E(x,y)∼DπϕRL[rθ(x,y)−βlog⁡πϕRL(y∣x)πSFT(y∣x)]+γ Ex∼Dpretrain[log⁡πϕRL(x)]\text{objective}(\phi) = \mathbb{E}_{(x,y)\sim D_{\pi^{RL}_\phi}}\left[r_\theta(x,y) - \beta\log\frac{\pi^{RL}_\phi(y\mid x)}{\pi^{SFT}(y\mid x)}\right] + \gamma\,\mathbb{E}_{x\sim D_{\text{pretrain}}}\left[\log \pi^{RL}_\phi(x)\right]

The first term is the reward minus the KL penalty, the same as in Stiennon et al.; the second is the log-likelihood on pretraining data, with the coefficient γ\gamma controlling its weight. This version is called PPO-ptx, and unless stated otherwise, "InstructGPT" in the paper refers to it.

The actual settings in the appendix are β=0.02\beta=0.02 and γ=27.8\gamma=27.8, with 256,000 training episodes in total covering about 31,000 distinct prompts; the number of pretraining examples is 8 times the number of RL episodes. The authors also compared this against simply raising the KL coefficient. That clearly lowers the reward on the validation set, and the DROP and SQuAD scores never come back. Mixing in pretraining gradients, on the other hand, pulls both regressions back with almost no change in reward.

Discussion

Main result: human preference

Original Figure 1. Win rate of each model against the 175B SFT model on the API prompt distribution, as model size changes (1.3B, 6B, 175B). GPT is about 0.14 to 0.24, GPT (prompted) about 0.23 to 0.41, SFT about 0.35 to 0.50, PPO about 0.52 to 0.63, and PPO-ptx about 0.52 to 0.66. Error bars are 95% confidence intervals.
Original Figure 1, Ouyang et al., from PDF page 2, arXiv license, scholarly quotation.

The vertical axis is the win rate against the 175B SFT model, chosen as the baseline because its performance sits roughly in the middle. From bottom to top: the original GPT-3, GPT-3 with a carefully designed few-shot prefix, SFT, PPO and PPO-ptx. That prefix won a little contest. Two authors each spent an hour interacting with GPT-3 to see who could find the prefix that scored highest with the reward model, and this was the winner (a footnote even makes a point of noting that Diogo Almeida won).

Each step up raises the win rate noticeably. The 1.3B PPO-ptx (about 0.52) beats the 175B original GPT-3 (about 0.24), which is where "a hundred times smaller and still winning" comes from. In direct head-to-head comparisons, the 175B InstructGPT wins 85% ± 3% of the time against the original 175B GPT-3, and 71% ± 4% against GPT-3 with the few-shot prefix. PPO-ptx and PPO get about the same preference, so mixing in pretraining gradients costs nothing in human preference.

Evaluate on "prompts originally submitted to GPT-3" instead, and the conclusion is the same. The held-out labelers, who contributed no training data, also gave preferences consistent with the training labelers. The authors additionally split the labelers into 5 groups for cross-validation. The reward model predicts the preferences of the held-out group with 69.6% accuracy, against 72.4% for the training groups.

More specific behaviors

Original Figure 4. Four labeler ratings on the API prompt distribution, comparing GPT, GPT (prompted), SFT, PPO and PPO-ptx. The share of responses that attempt the correct instruction rises from about 0.68 to about 0.93; the share that follow explicit constraints in the instruction rises from about 0.22 to about 0.43; the hallucination rate on closed-domain tasks is about 0.44 for GPT, about 0.28 for prompted GPT, about 0.08 for SFT, about 0.21 for PPO and about 0.16 for PPO-ptx; the share using language appropriate for a customer assistant rises from about 0.81 to about 0.90.
Original Figure 4, Ouyang et al., from PDF page 12, arXiv license, scholarly quotation.

For each response, labelers also filled in a checklist of specific items. Compared with GPT-3, the PPO models more often attempt the correct instruction, more often follow explicit constraints in the instruction (such as "in two paragraphs or less"), use language better suited to a customer-assistant context, and make up less content that isn't in the source on closed-domain tasks such as summarization.

The third panel deserves a second look. The lowest hallucination rate actually belongs to SFT (about 8%). PPO and PPO-ptx fall between 16% and 21%, still far below the original GPT-3's roughly 44%. The "21% vs. 41%" figure in the introduction compares InstructGPT with GPT-3.

Public datasets: truthfulness, toxicity and bias

Truthfulness. On TruthfulQA, PPO models produce "truthful and informative" answers at a rate slightly but significantly higher than GPT-3, without being specially instructed to tell the truth. The exception is the 1.3B PPO-ptx, which does slightly worse than GPT-3 of the same size. When the prompt asks the model to "answer I have no comment when uncertain," PPO models lean toward honestly declining to answer rather than confidently saying something wrong.

Toxicity. On RealToxicityPrompts, when the prompt asks for a respectful response, InstructGPT is about 25% less toxic than GPT-3; without that request, the two are about the same. When explicitly asked to produce toxic content, InstructGPT is in fact far more toxic than GPT-3.

Bias. On Winogender and CrowS-Pairs, InstructGPT is no less biased than GPT-3. When asked to behave respectfully, the model becomes more certain of its choices, and by the authors' entropy metric, its bias actually goes up.

Alignment tax. PPO-ptx recovers most of the regressions, and on HellaSwag it even beats GPT-3, but it still trails on DROP, SQuADv2 and translation.

The gap between public NLP datasets and real use

For comparison, the authors also took two public instruction datasets, FLAN and T0, sampled about 1 million examples from each, and used them to fine-tune the 175B GPT-3. On API prompts, these two models do better than the original GPT-3 and about as well as GPT-3 with the few-shot prefix, but worse than the SFT model trained only on labeler demonstrations. InstructGPT's outputs are preferred 78% ± 4% of the time over the FLAN version and 79% ± 4% over the T0 version.

The authors offer two explanations. Public NLP datasets lean toward classification and question answering, which are easy to score automatically, yet these two categories make up only about 18% of API usage, while open-ended generation and brainstorming make up about 57%. Public datasets also have a hard time matching the input diversity of real users.

Simple mistakes it still makes

The authors list several kinds of mistakes InstructGPT still makes:

  1. Accepting false premises. Ask "Why is it important to eat socks after meditating?" and it will earnestly list several "theories," for example that eating socks helps the brain come out of a meditative state.
  2. Excessive hedging. Ask what happens if you fire a cannonball at a pumpkin at high speed, and it first says there are too many factors to predict accurately before listing a few possibilities. The authors suspect this is because they told labelers to reward humble answers, and the reward model learned that "holding back a little" scores points.
  3. Multiple constraints. When an instruction carries several constraints at once, such as "list 10 movies made in the 1930s set in France," performance drops.

Another observation is more positive. The fine-tuning data is almost entirely English, with very few code-related instructions, yet InstructGPT can answer questions about code and follow instructions in languages other than English, though it often answers in English. The authors take this as a sign that "following instructions" generalizes, to some extent, to areas that were never directly supervised.

Cost, and "aligned to whom"

On cost, training the 175B SFT model takes 4.9 petaflop/s-days and the 175B PPO-ptx model takes 60, while training GPT-3 itself took 3,640. The authors conclude that, on their customers' task distribution, investing in aligning existing models is more cost-effective than making models 100 times bigger.

The paper devotes an entire section to "who are we aligning to?" There are four layers to what the model is actually aligned with. The first is the preferences of about 40 labelers, mostly English-speaking people living in the United States or Southeast Asia, who agree with one another about 73% of the time. The second is the researchers, who wrote the labeling instructions and answered edge cases in a chat room. The third is OpenAI API customers, who supplied the training prompts; those customers were originally picked off a waitlist whose initial seeds were OpenAI employees. The fourth is what customers consider "useful," and customer goals can conflict with those of end users; a customer might, for example, want users to stay on their platform as long as possible. The authors state plainly that they are not claiming the researchers, the labelers or the API customers are the "right" source of preferences.

The biggest limitation, though, is that in most cases the model does what the user tells it to, even when that could cause harm in the real world.

Conclusion

Every part of InstructGPT is old. The reward model dates from 2017, and PPO with a KL penalty comes from the 2020 summarization work. What's new is the setting, the many kinds of instructions real users throw at the API, and there a 1.3B model ends up preferred over the 175B GPT-3.

If you take one thing away from this paper, it is probably where the data comes from. An SFT model trained on a little over ten thousand labeler demonstrations beats the versions fine-tuned on a million examples of public NLP data. If you want a model to do well on what users actually ask, you first have to collect the questions users actually ask.

The excessive hedging the paper worries about still turns up a year later in the GPT-4 technical report. Asked where to find cheap cigarettes, an early version of GPT-4 opens with "As an AI model developed by OpenAI, I am programmed to not provide information on how to obtain illegal or harmful products."