Back to paper library
arXiv 2024Language & TransformersPaper notes

Tülu 3

The whole recipe, in the open

Original paper

Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Paper authors

Nathan Lambert · Jacob Morrison · Valentina Pyatkin …

Show all 23 authors

Nathan Lambert · Jacob Morrison · Valentina Pyatkin · Shengyi Huang · Hamish Ivison · Faeze Brahman · Lester James V. Miranda · Alisa Liu · Nouha Dziri · Shane Lyu · Yuling Gu · Saumya Malik · Victoria Graf · Jena D. Hwang · Jiangjiang Yang · Ronan Le Bras · Oyvind Tafjord · Chris Wilhelm · Luca Soldaini · Noah A. Smith · Yizhong Wang · Pradeep Dasigi · Hannaneh Hajishirzi

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2411.15124
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformerstulu32024,
  title = {{Tulu 3: Pushing Frontiers in Open Language Model Post-Training}},
  author = {Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and Lester James V. Miranda and Alisa Liu and Nouha Dziri and Shane Lyu and Yuling Gu and Saumya Malik and Victoria Graf and Jena D. Hwang and Jiangjiang Yang and Ronan Le Bras and Oyvind Tafjord and Chris Wilhelm and Luca Soldaini and Noah A. Smith and Yizhong Wang and Pradeep Dasigi and Hannaneh Hajishirzi},
  year = {2024},
  eprint = {2411.15124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.15124}
}
On this page

Pretrained model weights are released more and more often, but post-training is mostly shrouded in fog.

Llama 3's technical report says SFT, rejection sampling, and DPO ran for six rounds, but outsiders cannot see what data was used or how it was filtered. According to a footnote in the Tülu 3 report, as of November 2024 not one of the top 50 models on LMSYS Chatbot Arena had released its post-training data.

The open community's own post-training recipes, such as Tülu 2 and Zephyr-β, hold up on chat evaluations but clearly lag on math (MATH, GSM8K) and precise instruction following (IFEval).

In November 2024, Lambert and colleagues at the Allen Institute for AI (AI2) and the University of Washington published Tülu 3 to fill this gap. A tülu is a hybrid of a dromedary and a Bactrian camel, and the authors carried the name on to the third generation. This time, besides the model weights, they released the training data, data-generation pipelines, training code, evaluation tools, and intermediate checkpoints.

This note follows its three-stage recipe: SFT, DPO, and RLVR, a new method the authors propose.

Defining the problem

Original Figure 1. An overview of the Tülu 3 recipe. On the left is prompt curation, including public datasets, persona-driven synthetic instructions, and decontamination; below it are the core skills to strengthen: knowledge, reasoning, math, coding, chat, and safety. In the middle are three training stages, going from the base model through Tülu3-SFT and Tülu3-DPO to the final Tülu3, each marked above with the data it uses: data mixtures, on-policy and off-policy preference data, and prompts with verifiable rewards. At the bottom is the evaluation suite: development evaluations are used repeatedly at every stage, while unseen evaluations are used only at the end.
Original Figure 1, Lambert et al., from PDF page 5, CC BY 4.0.

Start reading this figure from the lower left. The authors first set out the core skills to strengthen: knowledge, reasoning, math, coding, instruction following and chat, and safety. Each skill maps to several evaluations.

The evaluations come in two groups. The development suite is used repeatedly throughout research and development, to select data and tune hyperparameters. The unseen suite is evaluated only once at the end, to confirm that the model has not overfit to the development suite. For knowledge, for example, the development evaluations are MMLU, PopQA, and TruthfulQA, and the unseen evaluations are MMLU-Pro and GPQA; for instruction following, the development evaluations are IFEval and AlpacaEval 2, and the unseen evaluations are IFEval-OOD and HREF.

The base models are Llama 3.1 8B and 70B, with 405B added later. Using only public or publishable data, can we build post-trained models on par with Llama 3.1 Instruct, or even better? That is the question the whole recipe sets out to answer.

Solving the problem

Data: fish out the exam questions first

Prompts come from two kinds of sources. One is existing public datasets, such as WildChat's real user conversations; the other is synthetic. The synthetic data uses an interesting method. First prepare a large number of "personas," then have GPT-4o role-play these personas and write questions, which increases the diversity of the questions. About 220,000 math and 35,000 code examples were synthesized.

The biggest risk with public datasets is that evaluation questions are mixed into them. The authors built a decontamination pipeline for this:

  1. Match only the prompts. Responses in training data are often regenerated, so matching responses is meaningless.
  2. Match with 8-grams. For each token in an evaluation, if some 8-gram containing it also appears in a training example, the token counts as matched. If more than 50% of the tokens in an evaluation question match the same training example, the two are judged to overlap. The authors tried full-string matching and embedding matching. The former is too strict, and the latter has a hard time telling "paraphrased" from "merely on a similar topic"; 8-grams catch cases such as "a math problem with only the numbers changed."
  3. Whether to keep a whole dataset. A training dataset counts as contaminated if it overlaps with more than 2% of the questions in any single evaluation. Datasets that overlap with the unseen suite are deleted entirely. For those that overlap with the development suite, the whole dataset is deleted if that does not hurt performance; otherwise only the overlapping examples are removed.

This pipeline really did catch quite a lot. For example, 11.3% of the problems in the widely used NuminaMath-TIR math dataset overlapped with the MATH evaluation; they were removed and the dataset was re-released.

SFT: the data mix and a loss-function trap

The key to SFT is the mix. Starting from the Tülu 2 mix, the authors added new data round after round, ran ablations, and then went through several rounds of decontamination to reach the final Tülu 3 SFT mix. Decontamination slightly lowers development-suite scores, an expected cost.

Training runs for 2 epochs with batch size 128 and a maximum length of 4,096 tokens; the learning rate is 5×10−65\times10^{-6} for 8B and 2×10−62\times10^{-6} for 70B. 8B trains on 32 GPUs for 6 hours, and 70B on 64 GPUs for 50 hours.

This section has two findings that bear on model merging and on implementation.

Model soups did not beat the best single model. The authors trained several SFT models with different random seeds and averaged them linearly with MergeKit. For 8B's five seeds, average scores ranged from 59.8 to 60.1, and the best soup (an average of two seeds) reached 60.2. 70B's three seeds scored 71.8, 70.0, and 72.6, and the best soup scored 72.5. The spread between seeds is substantial, 2.6 points for 70B, yet the soups did not consistently beat the best seed. In the end, the authors simply used the best single SFT model.

This differs from the conclusion of Model Soups on image classification, and from the way Llama 3 and Gemma 2 build averaging into their official pipelines. One possible reason is that only different seeds were averaged here, not different data or hyperparameters, so the models differ too little; the paper does not analyze this further.

The way the loss is averaged quietly changes the weighting. Early in training, the authors found that the same data trained in their own framework and on TPUs gave models that performed differently. They traced it all the way down to a problem in how the Transformers library computes the loss during gradient accumulation.

Let's work through the numbers. A batch has two examples. The first has 10 valid tokens and a total loss of 20; the second has 90 valid tokens and a total loss of 90.

  • When both go through the forward pass together, the library averages over all valid tokens at once: (20+90)/(10+90)=1.1(20+90)/(10+90)=1.1. Every token gets the same weight.
  • When they are sent in two passes with gradient accumulation, each pass averages on its own and the two results are then averaged: (20/10+90/90)/2=1.5(20/10+90/90)/2=1.5. Every example gets the same weight, and the tokens of the short example are amplified 9 times.

Change nothing but the number of gradient-accumulation steps, and the training objective changes with it. The authors' fix was to switch to a summed loss, no longer dividing by the number of tokens, and to retune the learning rate. This amounts to giving every token the same weight, and it also performed better in experiments. The issue was widely discussed in the community in the fall of 2024, and the paper cites the explanation by Unsloth and others.

Preference tuning: on-policy data and length-normalized DPO

Preference data generation improves on the UltraFeedback pipeline:

  1. Select prompts, including ones used in SFT, ones from the same sources but not used in SFT, and new prompts.
  2. Randomly pick four models from a pool of 22 to answer each prompt. This is off-policy data. Responses from the Tülu 3 SFT model itself are added too; this is on-policy data, which brings the preference data closer to the outputs the model being trained will actually produce.
  3. Ask GPT-4o to rate each response from 1 to 5 on four aspects: helpfulness, instruction following, truthfulness, and honesty. The response with the highest average score becomes "chosen," and one of the lower-scoring responses is randomly picked as "rejected."

In the ablation stage, 8B used 96,911 off-policy and 19,444 on-policy examples; the final preference data exceeds 270,000 examples. The authors also found that the more unique prompts the preference data contains, the better the performance; simply reusing the same prompts had limited effect.

On the algorithm side, the authors compared DPO, SimPO, PPO, and other methods, and finally chose length-normalized DPO:

max⁡πθ E[log⁡σ(β∣yc∣log⁡πθ(yc∣x)πref(yc∣x)−β∣yr∣log⁡πθ(yr∣x)πref(yr∣x))]\max_{\pi_\theta}\ \mathbb{E}\left[\log\sigma\left(\frac{\beta}{|y_c|}\log\frac{\pi_\theta(y_c\mid x)}{\pi_{\text{ref}}(y_c\mid x)}-\frac{\beta}{|y_r|}\log\frac{\pi_\theta(y_r\mid x)}{\pi_{\text{ref}}(y_r\mid x)}\right)\right]

The only difference from the original DPO is that each of the two log-probability ratios is divided by the length of its response, ∣yc∣|y_c| and ∣yr∣|y_r|. Both humans and model judges prefer long responses, and without normalization the model easily learns that "writing more is enough." The authors also chose it because it is simpler and faster than PPO, which makes it easy to iterate on the data mix.

RLVR: points only for right answers

The third stage is RLVR (Reinforcement Learning with Verifiable Rewards), proposed by the authors.

Original Figure 18. The RLVR flow. A prompt s_i from the training data is fed to the policy model π_θ, which produces a response a_i; the response goes to a verifiable reward function, which gives γ points if it is correct and 0 if it is wrong, and this scalar reward r_i is sent back to update the policy model.
Original Figure 18, Lambert et al., from PDF page 31, CC BY 4.0.

Standard RLHF uses a reward model to assign scores. RLVR replaces the reward model with a deterministic check function:

max⁡πθ Ey∼πθ(x)[v(x,y)−β KL[πθ(y∣x) ∥ πref(y∣x)]],v(x,y)={αif correct0otherwise\max_{\pi_\theta}\ \mathbb{E}_{y\sim\pi_\theta(x)}\Big[v(x,y)-\beta\,\mathrm{KL}\big[\pi_\theta(y\mid x)\,\|\,\pi_{\text{ref}}(y\mid x)\big]\Big],\qquad v(x,y)=\begin{cases}\alpha & \text{if correct}\\ 0 & \text{otherwise}\end{cases}

The main text writes the reward as α\alpha and the figure writes it as γ\gamma; they are the same value. The authors set it to 10 based on preliminary experiments and did not tune it further.

The check function is used only on three easily verifiable tasks. For the GSM8K and MATH training sets, the final answer is extracted and compared with the reference answer. In addition, instructions are taken from Tülu 2's SFT data and given IFEval-style format constraints (for example, "answer in no more than 100 words"), and a program checks whether they are followed. That is about 30,000 prompts in total.

Optimization uses PPO, and the authors list several implementation details that keep PPO stable: initialize the value model from a general reward model, turn off dropout during training, deduct 10 points when a response does not end properly, and normalize the advantages. On the infrastructure side, responses are generated with vLLM on dedicated GPUs while training runs on another set of GPUs, the two proceeding asynchronously. The policy always trains on the second-most-recent batch of generated data, striking a balance between efficiency and reproducibility.

Discussion

How much does RLVR add?

Table 23 of the paper gives the results of starting RLVR from the best DPO model:

8B DPO8B RLVR70B DPO70B RLVR
Average64.464.875.976.0
MATH42.043.762.363.0
GSM8K84.387.693.593.5
IFEval81.182.482.683.2
Safety87.285.589.088.3

On 8B, all three target tasks improve, with GSM8K up 3.3 points; but the safety score drops 1.7 points, and the overall average rises only 0.4. The gains for 70B are smaller still, with GSM8K already close to saturation. The authors also admit that some 8B runs could push GSM8K to 89.4 and IFEval to 84.8, but other metrics got worse and the overall average actually went down.

The training curves show that RLVR does keep raising accuracy on the training set, but the larger the KL divergence, the lower the average score tends to be. In other words, the farther the model moves from its starting point, the greater the risk of over-optimization.

405B was handled somewhat differently. GSM8K was already saturated after SFT and DPO, and IFEval data did not help much in early experiments, so only the MATH training set was used. After just 25 steps, MATH rose by more than 5 points; because of compute limits, training only reached 75 steps, and the curve had not yet saturated. 405B used 256 GPUs, 16 running vLLM for generation and 240 for training. Each round took about 550 seconds for generation, about 25 seconds to synchronize weights, and about 1,500 seconds for training. To save compute, the value model was only 8B.

Overall results

On the evaluation suite the authors put together, Tülu 3 8B averages 65.1, above Llama 3.1 8B Instruct's 62.9 but slightly below Qwen 2.5 7B Instruct's 66.5. Tülu 3 70B scores 76.2, above Llama 3.1 70B Instruct's 74.1 and Qwen 2.5 72B Instruct's 72.8, and also above GPT-4o mini's 69.6 and Claude 3.5 Haiku's 75.3. The 405B version averages 80.0 excluding safety, against 79.0 for DeepSeek-V3 and 80.5 for GPT-4o, all in the same range.

Keep in mind that most of these evaluations belong to the development suite the authors chose themselves, which was also the target of repeated optimization throughout development; the unseen suite is designed precisely to hedge against this risk.

What didn't work

The authors devote a whole section to "insights from the unfruitful," which is uncommon in technical reports:

  1. Online DPO: at each step, sample two responses from the current policy, label the preference with a reward model, then update. On math, GSM8K barely improved, and MATH actually regressed.
  2. Rejection sampling: the method Llama 3 leans on heavily brought only limited gains here for the compute invested. The authors observed that the judge model has to be strong enough to pick out good responses, and that putting the original response into the candidate pool works much better than choosing only among newly generated responses.

Limitations

The training data consists mostly of short conversations, averaging only 2.4 turns, with most examples under 2,048 tokens; long context and multi-turn conversation are clear gaps. Preference data and some synthetic data were generated or scored by GPT-4o, so "fully open" refers to the data and pipeline being public; the data itself still depends on closed models.

Conclusion

Half of Tülu 3's value lies in the models and half in that complete recipe. The decontamination thresholds, the SFT mix, how losses are summed, how preference data is generated, which PPO details to watch: all of it is written out clearly, failed attempts included.

RLVR itself is very plain, with points for right answers and nothing for wrong ones. On math and format following it brings steady but modest gains. A few months later, DeepSeek-R1 used rule-based rewards to push reasoning much further, and "verifiable rewards" became a common practice in RL post-training.

If you take only one thing away from this report, go back and check how your own training code averages the loss during gradient accumulation.