Back to paper library
arXiv 2023NeurIPS 2023Model adaptationPaper notes

DPO

You are the reward model

Original paper

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Paper authors

Rafael Rafailov · Archit Sharma · Eric Mitchell · Stefano Ermon · Christopher D. Manning · Chelsea Finn

Publication
NeurIPS 2023Conference
First submitted to arXiv
arXiv:2305.18290
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modeltuningdpo2023,
  title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
  author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Stefano Ermon and Christopher D. Manning and Chelsea Finn},
  year = {2023},
  eprint = {2305.18290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18290}
}
On this page

By 2023, the RLHF pipeline had settled into a fixed routine. First comes supervised fine-tuning (SFT). Then you collect human preferences between pairs of responses and train a reward model. Finally, PPO pushes the language model toward high scores, while a KL penalty holds it back from drifting too far from the original model.

The pipeline works well; it's how ChatGPT was trained. But it's also a lot of trouble. PPO training has to keep several large models in memory at once, including the policy being trained, a frozen reference model, the reward model and usually a value function. Training also has to keep sampling from the policy, and with so many hyperparameters, it can easily go unstable.

In this paper, Rafailov, Sharma, Mitchell and colleagues at Stanford present a somewhat surprising result. The objective RLHF is trying to optimize can be solved directly, without reinforcement learning, using nothing more than a binary classification loss. There's no need to train a separate reward model either, because the language model itself already contains one implicitly.

Defining the problem

Let's start by writing RLHF out as equations.

Given a prompt xx, the SFT model produces two responses, and a labeler picks out the better one, ywy_w, and the worse one, yly_l. Assume these preferences are driven by a hidden true reward r∗r^* and follow the Bradley-Terry model, meaning the probability that "y1y_1 beats y2y_2" is determined by the difference between their rewards:

p∗(y1≻y2∣x)=exp⁡r∗(x,y1)exp⁡r∗(x,y1)+exp⁡r∗(x,y2)=σ(r∗(x,y1)−r∗(x,y2))p^*(y_1 \succ y_2 \mid x) = \frac{\exp r^*(x, y_1)}{\exp r^*(x, y_1) + \exp r^*(x, y_2)} = \sigma\big(r^*(x, y_1) - r^*(x, y_2)\big)

Step one trains the reward model rϕr_\phi by maximum likelihood:

LR(rϕ)=−E(x,yw,yl)[log⁡σ(rϕ(x,yw)−rϕ(x,yl))]\mathcal{L}_R(r_\phi) = -\mathbb{E}_{(x, y_w, y_l)}\big[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\big]

Step two has the policy πθ\pi_\theta go after high scores, without straying too far from the reference model πref\pi_{\text{ref}} (usually the SFT model itself):

max⁡πθ Ex, y∼πθ[rϕ(x,y)]−β DKL[πθ(y∣x) ∥ πref(y∣x)]\max_{\pi_\theta}\ \mathbb{E}_{x,\,y\sim\pi_\theta}\big[r_\phi(x, y)\big] - \beta\,\mathbb{D}_{\text{KL}}\big[\pi_\theta(y\mid x)\,\|\,\pi_{\text{ref}}(y\mid x)\big]

β\beta controls how tight the leash is. Language generation is discrete, so this objective can't be differentiated directly, which is why reinforcement learning algorithms such as PPO are needed.

So can these two steps be merged into one?

Solving the problem

The optimal policy has a closed form

The key is the KL-constrained maximization problem in step two. Earlier work had already shown that its optimal solution can be written down directly:

πr(y∣x)=1Z(x) πref(y∣x)exp⁡ ⁣(1βr(x,y)),Z(x)=∑yπref(y∣x)exp⁡ ⁣(1βr(x,y))\pi_r(y\mid x) = \frac{1}{Z(x)}\,\pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{1}{\beta}r(x, y)\Big),\qquad Z(x)=\sum_y \pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{1}{\beta}r(x, y)\Big)

In other words, you start from the reference model, multiply each response's probability by a factor that grows with its reward, and then normalize the whole thing.

If there's a closed-form solution, why does everyone still use PPO? Because Z(x)Z(x) has to sum over every possible response, which for a language model simply can't be computed.

Writing it the other way around

The authors turn the equation above around, take the log of both sides and express the reward in terms of the policy:

r(x,y)=βlog⁡πr(y∣x)πref(y∣x)+βlog⁡Z(x)r(x, y) = \beta\log\frac{\pi_r(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x)

Z(x)Z(x) is still there, but it depends only on the prompt xx, not on the response yy. And the Bradley-Terry model only looks at the difference in reward between two responses. Subtract two responses to the same prompt, and βlog⁡Z(x)\beta\log Z(x) cancels out:

p∗(yw≻yl∣x)=σ(βlog⁡π∗(yw∣x)πref(yw∣x)−βlog⁡π∗(yl∣x)πref(yl∣x))p^*(y_w \succ y_l \mid x) = \sigma\Big(\beta\log\frac{\pi^*(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta\log\frac{\pi^*(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big)

The implicit reward differs from the original reward only by βlog⁡Z(x)\beta\log Z(x). That constant is the same for every response to a given prompt, so it cancels when two responses' rewards are subtracted. The formula above therefore gives, from the policy alone, exactly the same preference probability as the original reward.

The DPO loss

Since the preference probability can be written with the policy alone, it can take the reward model's place in that maximum-likelihood step directly. Replace π∗\pi^* with the πθ\pi_\theta being trained:

LDPO(πθ;πref)=−E(x,yw,yl)[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}}(\pi_\theta;\pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l)}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big)\Big]

It has exactly the same form as the reward model's loss, except that rϕr_\phi has been replaced by βlog⁡πθπref\beta\log\frac{\pi_\theta}{\pi_{\text{ref}}}. Training the language model amounts to training a reward model, and the optimal policy for that reward model is the language model itself. That is what the paper's subtitle, "Your Language Model is Secretly a Reward Model," is getting at.

The authors also prove that this change of variables doesn't limit which reward functions can be represented. Any part that depends only on the prompt, like Z(x)Z(x), never affected the preference probability or the optimal policy in the first place, and every class of equivalent reward functions can be written in the form βlog⁡ππref\beta\log\frac{\pi}{\pi_{\text{ref}}}.

The implementation takes only a few lines, and the paper's appendix gives the PyTorch code outright. Let's try it on two preference pairs, where each response's log probability is the sum over the whole response:

python
import torch
import torch.nn.functional as F

def dpo_loss(pi_logps, ref_logps, yw_idxs, yl_idxs, beta):
    """Implementation from Appendix B of the paper"""
    pi_yw_logps, pi_yl_logps = pi_logps[yw_idxs], pi_logps[yl_idxs]
    ref_yw_logps, ref_yl_logps = ref_logps[yw_idxs], ref_logps[yl_idxs]
    pi_logratios = pi_yw_logps - pi_yl_logps
    ref_logratios = ref_yw_logps - ref_yl_logps
    losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios))
    rewards = beta * (pi_logps - ref_logps).detach()
    return losses, rewards

# Two preference pairs: response 0 beats 1, response 2 beats 3 (log probabilities of whole responses)
ref_logps = torch.tensor([-12.0, -11.0, -20.0, -21.0])
pi_logps = torch.tensor([-11.0, -12.0, -21.0, -18.0], requires_grad=True)
yw, yl = torch.tensor([0, 2]), torch.tensor([1, 3])

losses, rewards = dpo_loss(pi_logps, ref_logps, yw, yl, beta=0.1)
losses.sum().backward()
print(losses.detach().numpy().round(3), rewards.numpy().round(2))
print(pi_logps.grad.numpy().round(3))

# [0.598 0.913] [ 0.1 -0.1 -0.1  0.3]
# [-0.045  0.045 -0.06   0.06 ]

In the first pair, the policy already prefers the good response more than the reference model does. The implicit rewards are 0.1 vs. −0.1, the ranking is correct, and the loss is 0.598. In the second pair, the policy has instead pushed up the probability of the bad response 3. The implicit rewards are −0.1 vs. 0.3, the ranking is wrong, and the loss is 0.913.

The gradients point the same way in both pairs, pushing the good response's probability up and the bad one's down. The difference is in their size. The second pair's gradient is 0.06, the first pair's 0.045. Writing the gradient out shows why:

∇θLDPO=−β E[σ(r^θ(x,yl)−r^θ(x,yw))⏟the more misranked, the larger the weight(∇θlog⁡π(yw∣x)−∇θlog⁡π(yl∣x))]\nabla_\theta\mathcal{L}_{\text{DPO}} = -\beta\,\mathbb{E}\Big[\underbrace{\sigma\big(\hat r_\theta(x, y_l) - \hat r_\theta(x, y_w)\big)}_{\text{the more misranked, the larger the weight}}\big(\nabla_\theta\log\pi(y_w\mid x) - \nabla_\theta\log\pi(y_l\mid x)\big)\Big]

r^θ=βlog⁡πθπref\hat r_\theta = \beta\log\frac{\pi_\theta}{\pi_{\text{ref}}} is the implicit reward. The more badly the implicit reward misranks a pair, the larger that pair's weight; pairs that are already ranked correctly get a small weight. The authors also point out that a simplified version without this weight makes the language model degenerate, and the appendix has the experiment to back it up.

The whole pipeline comes down to two steps. Sample two responses for each prompt and have people label which one they prefer, then train directly with the DPO loss. In practice, it's more common to just take an existing preference dataset. The reference model should ideally be the SFT model that originally produced those responses. If you don't have it, first run supervised fine-tuning on the preferred responses in the preference data and use the result as the reference model.

There's hardly any hyperparameter tuning to do. The paper's defaults are β=0.1\beta=0.1 (0.50.5 for the summarization task), a batch size of 64, RMSprop, a learning rate of 10−610^{-6} and linear warmup over the first 150 steps.

Original Figure 1. On the left is RLHF, which trains a reward model from preference data by maximum likelihood, then uses reinforcement learning to update the language model policy in a loop of "sample completions, label rewards"; on the right is DPO, which gets the final language model directly from preference data by maximum likelihood.
Original Figure 1, Rafailov et al., from PDF page 2, CC BY 4.0.

Put the two sides next to each other and the difference is plain. RLHF's reward model and reinforcement learning loop are replaced wholesale by a single maximum-likelihood step.

Discussion

The authors compare methods on three tasks, with models of up to 6 billion parameters.

Same objective, so who optimizes it better?

The first task is controlled sentiment generation. Given the opening of an IMDb movie review, the model has to continue it with a positive review. The preferences here come from a pretrained sentiment classifier, so the "true reward" is known and different algorithms can be compared precisely. The model is GPT-2-large.

The RLHF objective wants a high score and closeness to the reference model at the same time, so a comparison can't look at the score alone. It has to look at the trade-off between reward and KL divergence. The authors swept different hyperparameters for each method, 22 training runs in total.

Original Figure 2. Left: expected reward against KL divergence at different stages of training for each method on the IMDb sentiment generation task. DPO (yellow) has the highest reward at every KL value and is already close to 1.0 at a KL of about 2.5; PPO and PPO-GT, which can read the true reward, need a larger KL to reach a lower reward; unlikelihood and preferred-FT do worse. Right: win rate of TL;DR summaries against human-written reference summaries, plotted against sampling temperature. DPO is about 0.61 at temperature 0 and PPO about 0.57, with PPO falling below 0.1 at high temperatures; SFT and preferred-FT are about 0.4, zero-shot GPT-J is under 0.1, and best of 128 is about 0.57 at temperature 0.5.
Original Figure 2, Rafailov et al., from PDF page 7, CC BY 4.0.

Start with the left plot. The horizontal axis is the KL divergence from the reference model, the vertical axis is reward, and the further up and to the left, the better. The yellow DPO pushes reward close to 1 while KL is still very small, and its entire frontier sits above the other methods. PPO optimizes the same objective but is clearly less efficient. Even PPO-GT, which can read the true reward directly, has a frontier that falls short of DPO's.

Summarization and dialogue

The right plot is the second task, Reddit TL;DR summarization, using the very human preferences that Stiennon et al. collected. All methods start from the same GPT-J SFT model, and GPT-4 is asked to judge whether the model's summary or the human-written reference summary is better.

At temperature 0, DPO's win rate is about 61%, while PPO, at its best temperature (also 0), is about 57%. The more obvious difference shows up as the temperature rises. PPO falls all the way to roughly the level of GPT-J without any fine-tuning, while DPO holds much steadier. The authors also ran a human evaluation, in which DPO (temperature 0.25) was preferred over PPO (temperature 0) 58% of the time.

The third task is single-turn dialogue from Anthropic HH. The data has about 170,000 dialogues, each ending with two responses and a human preference between them. There's no ready-made SFT model here, so the authors first fine-tune Pythia-2.8B on the preferred responses in the preference data to serve as the reference model, then run DPO.

Original Figure 3. Left: win rate of each method's responses against the chosen responses in the dataset for Anthropic HH single-turn dialogue, plotted against sampling temperature. DPO rises from about 0.37 at temperature 0.25 to about 0.63 at 1.0; best of 128 is about 0.54 to 0.61; preferred-FT about 0.3 to 0.43; Pythia-2.8B with a 2-shot prompt about 0.16 to 0.25. Right: DPO's win rate over fine-tuning steps at temperatures 1.0 and 0.7, moving up and down roughly between 0.5 and 0.64 after about 300 steps.
Original Figure 3, Rafailov et al., from PDF page 8, CC BY 4.0.

The dashed line at 0.5 marks a tie with the chosen responses in the dataset. DPO is the only method that gets above this line without heavy sampling. Best of 128 gets above it too, but it has to draw 128 responses for every question and let the reward model pick one, which costs far more at inference time. The authors also tried a publicly available PPO model and couldn't find any prompt or temperature that let it beat the base Pythia-2.8B. The right plot shows that DPO converges quickly, getting close to its best level within a few hundred steps.

Moving to unseen data

The authors took the models trained on TL;DR and used them as they were to summarize CNN/DailyMail news articles, to see whether they had only learned what Reddit looks like. Judged by GPT-4 on win rate against human-written summaries, DPO scores 0.36 and PPO 0.26 at temperature 0; at temperature 0.25, it's 0.31 vs. 0.23. DPO still leads, and that's with PPO having used extra unlabeled Reddit prompts during training.

Can GPT-4 be trusted as a judge?

All of these experiments rely on GPT-4 as the judge, so the authors also checked how well it agrees with humans. With the simplest prompt, which asks "which summary better covers the important points," GPT-4 prefers long, repetitive summaries more than humans do. With a prompt that also asks for conciseness, GPT-4's judgments come closer to the humans'. Across the three comparisons, GPT-4 agrees with humans 67% to 86% of the time, while humans agree with each other 65% to 87% of the time, about the same.

Questions still open

The authors list several themselves:

  • When DPO policies move to unseen data, can they keep performing as well as methods trained with an explicit reward model? The news summarization experiment above is only preliminary evidence.
  • What form does reward over-optimization take in DPO? In the dialogue experiment, DPO's win rate dips slightly late in training. Could that be one example?
  • The experiments only go up to 6B. Does DPO scale to models several orders of magnitude larger?
  • GPT-4's ratings shift with the prompt. How do you get reliable judgments out of an automated judge?

What came next

DPO was quickly adopted on a large scale. Llama 3's post-training in 2024 consisted of several rounds of SFT plus DPO. Meta had also compared it against PPO and concluded that DPO needed less compute for large models and performed better on instruction following. Tülu 3, from the same year, placed DPO after SFT as part of its fully open post-training recipe.

There were dissenting voices as well. The 2024 paper Is DPO Superior to PPO for LLM Alignment? argues that DPO may find biased solutions that exploit responses outside the distribution of the preference data, and that its performance drops noticeably when the model's own outputs differ a lot from the distribution of the preference data. Once its authors tuned PPO properly, PPO beat DPO on both dialogue and code generation, with the biggest gap on the hardest competitive programming problems.

Then in 2025, reasoning models invited reinforcement learning back, only this time the reward often came from answers that could be checked automatically rather than from human preferences.

Conclusion

At its core, DPO is a single change of variables. The optimal policy under a KL constraint has a closed form. Turn it around to express the reward in terms of the policy, and the intractable partition function cancels out in the Bradley-Terry subtraction, leaving RLHF as a binary classification loss. No reward model to train, no sampling during training, and none of PPO's pile of hyperparameters.

If you have pairwise preference data on hand, DPO should be the first method you try. Just remember to keep the reference model as close as possible to the model that generated that data. If the distributions are too far apart, its performance takes a hit.

The paper said reinforcement learning could be done without; a year later, Llama 3's post-training really did swap PPO for DPO.