Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Publication
- NeurIPS 2023Conference
- First submitted to arXiv
- arXiv:2305.18290
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modeltuningdpo2023,
title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Stefano Ermon and Christopher D. Manning and Chelsea Finn},
year = {2023},
eprint = {2305.18290},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.18290}
}
By 2023, the RLHF pipeline had settled into a fixed routine. First comes supervised fine-tuning (SFT). Then you collect human preferences between pairs of responses and train a reward model. Finally, PPO pushes the language model toward high scores, while a KL penalty holds it back from drifting too far from the original model.
The pipeline works well; it's how ChatGPT was trained. But it's also a lot of trouble. PPO training has to keep several large models in memory at once, including the policy being trained, a frozen reference model, the reward model and usually a value function. Training also has to keep sampling from the policy, and with so many hyperparameters, it can easily go unstable.
In this paper, Rafailov, Sharma, Mitchell and colleagues at Stanford present a somewhat surprising result. The objective RLHF is trying to optimize can be solved directly, without reinforcement learning, using nothing more than a binary classification loss. There's no need to train a separate reward model either, because the language model itself already contains one implicitly.
Defining the problem
Let's start by writing RLHF out as equations.
Given a prompt , the SFT model produces two responses, and a labeler picks out the better one, , and the worse one, . Assume these preferences are driven by a hidden true reward and follow the Bradley-Terry model, meaning the probability that " beats " is determined by the difference between their rewards:
Step one trains the reward model by maximum likelihood:
Step two has the policy go after high scores, without straying too far from the reference model (usually the SFT model itself):
controls how tight the leash is. Language generation is discrete, so this objective can't be differentiated directly, which is why reinforcement learning algorithms such as PPO are needed.
So can these two steps be merged into one?
Solving the problem
The optimal policy has a closed form
The key is the KL-constrained maximization problem in step two. Earlier work had already shown that its optimal solution can be written down directly:
In other words, you start from the reference model, multiply each response's probability by a factor that grows with its reward, and then normalize the whole thing.
If there's a closed-form solution, why does everyone still use PPO? Because has to sum over every possible response, which for a language model simply can't be computed.
Writing it the other way around
The authors turn the equation above around, take the log of both sides and express the reward in terms of the policy:
is still there, but it depends only on the prompt , not on the response . And the Bradley-Terry model only looks at the difference in reward between two responses. Subtract two responses to the same prompt, and cancels out:
The implicit reward differs from the original reward only by . That constant is the same for every response to a given prompt, so it cancels when two responses' rewards are subtracted. The formula above therefore gives, from the policy alone, exactly the same preference probability as the original reward.
The DPO loss
Since the preference probability can be written with the policy alone, it can take the reward model's place in that maximum-likelihood step directly. Replace with the being trained:
It has exactly the same form as the reward model's loss, except that has been replaced by . Training the language model amounts to training a reward model, and the optimal policy for that reward model is the language model itself. That is what the paper's subtitle, "Your Language Model is Secretly a Reward Model," is getting at.
The authors also prove that this change of variables doesn't limit which reward functions can be represented. Any part that depends only on the prompt, like , never affected the preference probability or the optimal policy in the first place, and every class of equivalent reward functions can be written in the form .
The implementation takes only a few lines, and the paper's appendix gives the PyTorch code outright. Let's try it on two preference pairs, where each response's log probability is the sum over the whole response:
import torch
import torch.nn.functional as F
def dpo_loss(pi_logps, ref_logps, yw_idxs, yl_idxs, beta):
"""Implementation from Appendix B of the paper"""
pi_yw_logps, pi_yl_logps = pi_logps[yw_idxs], pi_logps[yl_idxs]
ref_yw_logps, ref_yl_logps = ref_logps[yw_idxs], ref_logps[yl_idxs]
pi_logratios = pi_yw_logps - pi_yl_logps
ref_logratios = ref_yw_logps - ref_yl_logps
losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios))
rewards = beta * (pi_logps - ref_logps).detach()
return losses, rewards
# Two preference pairs: response 0 beats 1, response 2 beats 3 (log probabilities of whole responses)
ref_logps = torch.tensor([-12.0, -11.0, -20.0, -21.0])
pi_logps = torch.tensor([-11.0, -12.0, -21.0, -18.0], requires_grad=True)
yw, yl = torch.tensor([0, 2]), torch.tensor([1, 3])
losses, rewards = dpo_loss(pi_logps, ref_logps, yw, yl, beta=0.1)
losses.sum().backward()
print(losses.detach().numpy().round(3), rewards.numpy().round(2))
print(pi_logps.grad.numpy().round(3))
# [0.598 0.913] [ 0.1 -0.1 -0.1 0.3]
# [-0.045 0.045 -0.06 0.06 ]
In the first pair, the policy already prefers the good response more than the reference model does. The implicit rewards are 0.1 vs. −0.1, the ranking is correct, and the loss is 0.598. In the second pair, the policy has instead pushed up the probability of the bad response 3. The implicit rewards are −0.1 vs. 0.3, the ranking is wrong, and the loss is 0.913.
The gradients point the same way in both pairs, pushing the good response's probability up and the bad one's down. The difference is in their size. The second pair's gradient is 0.06, the first pair's 0.045. Writing the gradient out shows why:
is the implicit reward. The more badly the implicit reward misranks a pair, the larger that pair's weight; pairs that are already ranked correctly get a small weight. The authors also point out that a simplified version without this weight makes the language model degenerate, and the appendix has the experiment to back it up.
The whole pipeline comes down to two steps. Sample two responses for each prompt and have people label which one they prefer, then train directly with the DPO loss. In practice, it's more common to just take an existing preference dataset. The reference model should ideally be the SFT model that originally produced those responses. If you don't have it, first run supervised fine-tuning on the preferred responses in the preference data and use the result as the reference model.
There's hardly any hyperparameter tuning to do. The paper's defaults are ( for the summarization task), a batch size of 64, RMSprop, a learning rate of and linear warmup over the first 150 steps.

Put the two sides next to each other and the difference is plain. RLHF's reward model and reinforcement learning loop are replaced wholesale by a single maximum-likelihood step.
Discussion
The authors compare methods on three tasks, with models of up to 6 billion parameters.
Same objective, so who optimizes it better?
The first task is controlled sentiment generation. Given the opening of an IMDb movie review, the model has to continue it with a positive review. The preferences here come from a pretrained sentiment classifier, so the "true reward" is known and different algorithms can be compared precisely. The model is GPT-2-large.
The RLHF objective wants a high score and closeness to the reference model at the same time, so a comparison can't look at the score alone. It has to look at the trade-off between reward and KL divergence. The authors swept different hyperparameters for each method, 22 training runs in total.

Start with the left plot. The horizontal axis is the KL divergence from the reference model, the vertical axis is reward, and the further up and to the left, the better. The yellow DPO pushes reward close to 1 while KL is still very small, and its entire frontier sits above the other methods. PPO optimizes the same objective but is clearly less efficient. Even PPO-GT, which can read the true reward directly, has a frontier that falls short of DPO's.
Summarization and dialogue
The right plot is the second task, Reddit TL;DR summarization, using the very human preferences that Stiennon et al. collected. All methods start from the same GPT-J SFT model, and GPT-4 is asked to judge whether the model's summary or the human-written reference summary is better.
At temperature 0, DPO's win rate is about 61%, while PPO, at its best temperature (also 0), is about 57%. The more obvious difference shows up as the temperature rises. PPO falls all the way to roughly the level of GPT-J without any fine-tuning, while DPO holds much steadier. The authors also ran a human evaluation, in which DPO (temperature 0.25) was preferred over PPO (temperature 0) 58% of the time.
The third task is single-turn dialogue from Anthropic HH. The data has about 170,000 dialogues, each ending with two responses and a human preference between them. There's no ready-made SFT model here, so the authors first fine-tune Pythia-2.8B on the preferred responses in the preference data to serve as the reference model, then run DPO.

The dashed line at 0.5 marks a tie with the chosen responses in the dataset. DPO is the only method that gets above this line without heavy sampling. Best of 128 gets above it too, but it has to draw 128 responses for every question and let the reward model pick one, which costs far more at inference time. The authors also tried a publicly available PPO model and couldn't find any prompt or temperature that let it beat the base Pythia-2.8B. The right plot shows that DPO converges quickly, getting close to its best level within a few hundred steps.
Moving to unseen data
The authors took the models trained on TL;DR and used them as they were to summarize CNN/DailyMail news articles, to see whether they had only learned what Reddit looks like. Judged by GPT-4 on win rate against human-written summaries, DPO scores 0.36 and PPO 0.26 at temperature 0; at temperature 0.25, it's 0.31 vs. 0.23. DPO still leads, and that's with PPO having used extra unlabeled Reddit prompts during training.
Can GPT-4 be trusted as a judge?
All of these experiments rely on GPT-4 as the judge, so the authors also checked how well it agrees with humans. With the simplest prompt, which asks "which summary better covers the important points," GPT-4 prefers long, repetitive summaries more than humans do. With a prompt that also asks for conciseness, GPT-4's judgments come closer to the humans'. Across the three comparisons, GPT-4 agrees with humans 67% to 86% of the time, while humans agree with each other 65% to 87% of the time, about the same.
Questions still open
The authors list several themselves:
- When DPO policies move to unseen data, can they keep performing as well as methods trained with an explicit reward model? The news summarization experiment above is only preliminary evidence.
- What form does reward over-optimization take in DPO? In the dialogue experiment, DPO's win rate dips slightly late in training. Could that be one example?
- The experiments only go up to 6B. Does DPO scale to models several orders of magnitude larger?
- GPT-4's ratings shift with the prompt. How do you get reliable judgments out of an automated judge?
What came next
DPO was quickly adopted on a large scale. Llama 3's post-training in 2024 consisted of several rounds of SFT plus DPO. Meta had also compared it against PPO and concluded that DPO needed less compute for large models and performed better on instruction following. Tülu 3, from the same year, placed DPO after SFT as part of its fully open post-training recipe.
There were dissenting voices as well. The 2024 paper Is DPO Superior to PPO for LLM Alignment? argues that DPO may find biased solutions that exploit responses outside the distribution of the preference data, and that its performance drops noticeably when the model's own outputs differ a lot from the distribution of the preference data. Once its authors tuned PPO properly, PPO beat DPO on both dialogue and code generation, with the biggest gap on the hardest competitive programming problems.
Then in 2025, reasoning models invited reinforcement learning back, only this time the reward often came from answers that could be checked automatically rather than from human preferences.
Conclusion
At its core, DPO is a single change of variables. The optimal policy under a KL constraint has a closed form. Turn it around to express the reward in terms of the policy, and the intractable partition function cancels out in the Bradley-Terry subtraction, leaving RLHF as a binary classification loss. No reward model to train, no sampling during training, and none of PPO's pile of hyperparameters.
If you have pairwise preference data on hand, DPO should be the first method you try. Just remember to keep the reference model as close as possible to the model that generated that data. If the distributions are too far apart, its performance takes a hit.
The paper said reinforcement learning could be done without; a year later, Llama 3's post-training really did swap PPO for DPO.