Learning to summarize from human feedback
- Publication
- NeurIPS 2020Conference
- First submitted to arXiv
- arXiv:2009.01325
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modeltuninglearningtosummarize2020,
title = {{Learning to summarize from human feedback}},
author = {Nisan Stiennon and Long Ouyang and Jeff Wu and Daniel M. Ziegler and Ryan Lowe and Chelsea Voss and Alec Radford and Dario Amodei and Paul Christiano},
year = {2020},
eprint = {2009.01325},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2009.01325}
}
The standard way to train a summarization model is supervised learning on human-written reference summaries, followed by scoring with ROUGE.
Both are only approximations of a “good summary.” Maximum likelihood asks the model to put probability on every human-written summary, including the badly written ones. It also can't tell important errors from unimportant ones, so making up a fact and picking the wrong synonym cost about the same in the loss. ROUGE counts word overlap with the reference summary, and its correlation with human judgment has long been criticized.
Stiennon, Ouyang, Wu, Ziegler and colleagues at OpenAI published this paper in September 2020. They had people compare two summaries directly, learned a reward model that predicts human preferences, and then optimized against it with reinforcement learning. The result is a 1.3-billion-parameter model that labelers prefer more often than a supervised model 10 times its size, and more often than the human-written reference summaries in the dataset.
The pipeline follows the “learning a reward from preferences” approach of Christiano et al. in 2017, with video clips swapped for text summaries; the team lead was Christiano himself. A year and a half later, InstructGPT used these three steps almost unchanged.
Defining the problem
The authors chose summarization as their test task because it has a long research history, and its quality is so subjective that it's hard to measure without human judgment.
The dataset is Reddit's TL;DR, about 3 million posts, each with a “too long; didn't read” summary written by the original poster. The authors filtered out subreddits the general public wouldn't understand and kept only posts whose reference summaries are between 24 and 48 tokens long, to minimize the effect of length on quality judgments, leaving 123,169 posts. (This filtered dataset later became a go-to benchmark for RLHF; the summarization experiments in the DPO paper three years later use it too.)
They deliberately avoided the more common CNN/DailyMail news summarization. On CNN/DM, the lead-3 baseline, which simply takes the first three sentences of the article, is already strong; labelers even prefer lead-3 to the dataset's reference summaries. T5 with low-temperature sampling already beats the reference summaries, while copying heavily from the article. Extractive baselines do poorly on TL;DR, which makes a model's real summarization ability easier to see.
The task is defined as producing a summary of under 48 tokens that is as good as possible. Quality means how faithfully a reader who reads only the summary, not the original post, can understand what the post is saying.
The year before, Ziegler et al. on the same team had already tried human feedback on TL;DR and CNN/DM, but ran into a problem. They reported that the “quality” the model learned was not what the researchers wanted. Summaries the labelers liked looked mediocre to the researchers, and the model ended up doing little more than copying from the original text. So how do you get human feedback to actually reflect the quality the researchers want?
Solving the problem
Three steps

- Collect human comparisons: for each post, sample summaries from the current policy, the initial policy, the reference summaries and various baselines, and send them in pairs to labelers, who pick the better one.
- Train a reward model: given a post and a summary, predict the log odds that a person picks this summary.
- Optimize the policy with PPO: use the reward model's output as the reward and train the summarization model with reinforcement learning.
These three steps can be repeated, with samples from the new policy sent back for human comparison. The authors also admit that in practice they didn't strictly cycle through rounds; instead, they kept adjusting data collection and training as labels accumulated.
All models are GPT-3-style Transformer decoders, with the main experiments at 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters. The starting point is a model fine-tuned with supervised learning on TL;DR, referred to below as the SFT model. It generates the initial comparison samples, initializes both the policy and the reward model, and serves as the baseline for comparison.
The reward model
The reward model starts from the SFT model with a randomly initialized linear layer added on top, which outputs a scalar , where is the post and the summary. If the person chose , the loss is:
This is the same Bradley–Terry loss as in the 2017 paper, except that the summed rewards of two trajectories are replaced by the scores of two summaries. is the sigmoid; the bigger the score gap between the two summaries, the more certain the model is about which one the person will pick. After training, the authors shift the outputs so that the reference summaries have an average score of 0.
The policy: PPO plus a KL penalty
The policy is also initialized from the SFT model. Each generated BPE token is one step, and a finished summary (once EOS is output) is one episode; the reward model scores the whole summary only once, at the end, and the discount factor is .
The key design choice is adding a KL penalty to the reward:
is the policy being trained and is the frozen SFT model. If the policy generates a summary that the SFT model considers very unlikely, the second term gets large and more points are deducted.
Let's work it out for a summary only 4 tokens long. Implementations commonly spread the KL penalty across tokens and add the reward model's score only at the last token:
import torch
beta = 0.05
rm_score = torch.tensor(2.0) # reward model score for the whole summary
logp_rl = torch.tensor([-1.2, -0.4, -2.0, -0.7]) # log prob of each token under the RL policy
logp_sft = torch.tensor([-1.5, -0.5, -3.1, -0.9]) # log prob of the same tokens under the SFT model
log_ratio = logp_rl - logp_sft # tensor([0.3000, 0.1000, 1.1000, 0.2000])
rewards = -beta * log_ratio # deduct the KL penalty at every token first
rewards[-1] += rm_score # reward model score added only at the last token
print(rewards) # tensor([-0.0150, -0.0050, -0.0550, 1.9900])
print(rewards.sum()) # tensor(1.9150)
The sum is the above. The third token is a word choice the RL policy favors more than the SFT model does, with a log ratio of 1.1, so it takes the biggest deduction. In the paper's main experiments, is 0.05.
The authors say the penalty does two things. First, like an entropy bonus, it encourages exploration and keeps the policy from collapsing onto a single way of writing. Second, it keeps the policy's outputs from straying too far from the summaries the reward model saw during training; the reward model has only learned from summaries near the SFT model, and once outputs drift too far from those, its scores can't be trusted.
PPO's value function is also a separate Transformer, sharing no parameters with the policy, and is initialized from the reward model's weights. The authors' ablation shows that with shared parameters, value function updates early in training damage the pretrained policy. The cost is that the RL stage has to fit an entire extra model in memory; the policy, the value function, the reward model and the SFT model used for the KL term are four models of the same size.
Label quality
The earlier work's problem lay in the labeling. The authors made two changes. First, they switched entirely to offline collection, sending labelers a large batch of comparisons at a time and, once it was all in, retraining on all the accumulated data, rather than collecting labels while training. Second, they worked closely with the labelers.
The researchers first did many comparisons themselves, discussed their disagreements, and wrote labeling guidelines. Labelers went through a paid training phase, and early on they immediately saw which summary the researchers had picked and why; only those who met thresholds for speed and agreement stayed on. During actual labeling, labelers first wrote down their “naive interpretation” of a summary without seeing the original post, then used a 9-point scale to say how confident they were that A was better than B. Everyone got 10% to 20% of their questions from a shared pool for calibration. The team ran a shared chat room to answer questions, occasionally had one-on-one video calls with labelers to talk through disagreements, and let go of the worst performers.
As a result, labelers agreed with researchers 77% ± 2% of the time, while researchers agreed with one another only 73% ± 4% of the time. The comparisons are inherently subjective and researchers often disagree among themselves; these numbers show that agreement between labelers and researchers had reached the level of agreement among the researchers themselves. The released dataset contains 64,832 summary comparisons.
Discussion
Main result: beating the reference summaries

The vertical axis is the fraction of the time labelers prefer the model's summary over the reference summary; the dashed line at 0.5 means a tie with the human-written summaries. The three lines are pretrain only (“zero-shot,” with examples stuffed into the prompt), supervised fine-tuning, and models trained with human feedback.
The 1.3B human feedback model is preferred 61% of the time, well ahead of a supervised model 10 times its size (12.9B, 43%). The 6.7B human feedback model rises further to about 70%, which shows the method also benefits from scale. Even scaled up to 12.9B, the supervised model still loses to the reference summaries.
Length is a confounder. The human feedback models learned to write longer summaries, and within the 48-token limit, writing a bit more usually covers more information. After the authors control for length with logistic regression, the human feedback models' preference rate against the reference summaries drops by about 5%, and the 6.7B model is still at about 65%. The figure caption also states that at 6.7B, length explains about a third of the gap between the human feedback and supervised models.
The authors also had labelers rate summaries on a 7-point scale along four dimensions: coverage, accuracy, coherence and overall quality. The human feedback models beat the supervised baseline on every dimension, with the biggest gap in coverage. 45% of the 6.7B PPO model's summaries get a perfect overall score of 7, against 20% for the supervised baseline and 23% for the reference summaries.
Transfer to news
The human feedback model trained only on Reddit was used to summarize CNN/DM news directly, without any news-specific fine-tuning. It clearly beats the supervised model trained on the same TL;DR data, and the 6.7B version nearly catches up with a 6.7B model fine-tuned with supervision directly on CNN/DM, while writing summaries only about half as long.
This comparison uses the 7-point scale, because the two models' length distributions barely overlap, which makes direct pairwise comparison hard. Grouping by length, the authors found that the human feedback model might score even higher if it wrote longer summaries.
What happens when you over-optimize
The reward model is only an approximation of labeler preferences. Its capacity is limited, and it has seen only a narrow range of summaries. So how far can you optimize against it before its scores stop being trustworthy?

Using an earlier version of the reward model, the authors trained a series of policies with different KL coefficients . The smaller is, the farther the policy moves from the SFT model, and the larger the KL on the horizontal axis. The dashed line is the reward model's own predicted preference rate, which keeps climbing until it nears 100% at the far right. The solid line is the labelers' actual preference. It rises under light optimization, peaks around a KL of 10, then starts sliding, nearly reaching zero at a KL of 250. In the end, the reward model is negatively correlated with human preference.
The appendix shows summaries from that KL-250 policy. A post by a 28-year-old man who wants to take up gymnastics gets summarized as “28yo dude stubbornly postponees start pursuing gymnastics hobby citing logistics reasons despite obvious interest???”, followed by a string of ungrammatical, profanity-laced sentences. Almost every summary listed in the appendix has the same phrase, “want change this ...”, popping up in it. The authors' assessment is that these summaries are long and strange but still roughly capture the gist.
This is exactly what the KL penalty is there to prevent. OpenAI's 2022 paper Scaling Laws for Reward Model Overoptimization later used a fixed “gold” reward model to stand in for humans and measured this curve systematically, finding that its shape depends on the optimization method while its coefficients vary smoothly with the reward model's parameter count. ROUGE has the same problem. When optimizing ROUGE with best-of-N sampling, quality peaks sooner than when optimizing against the reward model, and the peak is much lower.
What the reward model learned
The authors trained 7 reward models ranging from 160 million to 13 billion parameters, on 8k to 64k comparisons. Doubling the data raises validation accuracy by about 1.1%; doubling the model, by about 1.8%. The 6.7B reward model trained on all the data starts to approach the accuracy of a single labeler.
On CNN/DM, which they were never trained on, the 1.3B and 6.7B reward models agree with labelers 62.4% and 66.5% of the time, respectively; labelers agree with each other 66.9% of the time.
The reward model is also sensitive to details. When labelers were asked to make minimal edits that improve a summary, the 6.7B reward model picks the edited version 82.8% of the time, against 84.1% for a separate group of human evaluators. When the roles of two people in a reference summary are swapped, the 6.7B model picks the original, correct version 97.2% of the time.
Biases show up too. For improvements that make a summary shorter, the 6.7B reward model agrees only 62.6% of the time, against 76.4% for humans. The reward model prefers long summaries, which matches the earlier observation that the human feedback models tend to write long.
ROUGE, meanwhile, gets worse as the models get stronger. When comparing two samples from the supervised model, ROUGE agrees with labelers about 57% of the time; when comparing two samples from the human feedback model, that falls to about 50%, a coin flip. Using the supervised model's log probability as the metric fares no better, dropping to 50% or below on the human feedback model's samples, while the reward model still manages 62%.
The price
The most immediate limitation is cost. RL fine-tuning of the 6.7B model took about 320 GPU-days; the training set took labelers thousands of hours, plus a lot of researcher time to maintain quality. Because it was so expensive, the authors couldn't run an important control, namely supervised learning on an equal amount of high-quality human demonstrations, to see how that would compare.
In 2024, Huang et al. reproduced this pipeline from scratch, listing more than twenty key implementation details and using Pythia models to replicate the trend of quality improving with model scale. Their 2.8B and 6.9B models beat the 1.3B checkpoint OpenAI released.
Conclusion
The 1.3B human feedback model is preferred over a supervised model 10 times its size, and over the TL;DRs the Reddit posters wrote themselves. On tasks where you can't write down a scoring rubric, having people compare and then having the model chase those comparisons works.
If you want to do this yourself, the part most worth copying turns out to be the labeling process. Write guidelines first, calibrate first, keep a shared pool of questions, and keep an eye on agreement with the researchers; only then will the reward model learn the quality you want. The PPO part is much easier to handle by comparison.
And whatever you do, don't skimp on the KL penalty. Without it, the model quickly learns to write sentences the reward model loves and people can't make sense of.