Training Compute-Optimal Large Language Models
- Publication
- NeurIPS 2022Conference
- First submitted to arXiv
- arXiv:2203.15556
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformerschinchilla2022,
title = {{Training Compute-Optimal Large Language Models}},
author = {Jordan Hoffmann and Sebastian Borgeaud and Arthur Mensch and Elena Buchatskaya and Trevor Cai and Eliza Rutherford and Diego de Las Casas and Lisa Anne Hendricks and Johannes Welbl and Aidan Clark and Tom Hennigan and Eric Noland and Katie Millican and George van den Driessche and Bogdan Damoc and Aurelia Guy and Simon Osindero and Karen Simonyan and Erich Elsen and Jack W. Rae and Oriol Vinyals and Laurent Sifre},
year = {2022},
eprint = {2203.15556},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2203.15556}
}
We have run out of jokes about the authors’ unusual model names.
This paper is an advanced version of Scaling Laws. The authors challenge some previous viewpoints and propose a new training strategy.
Defining the Problem
At the paper’s publication in 2022, language models were rapidly growing in parameter count. Both dense Transformers and MoE models followed this trend, although total and active parameter counts are not directly interchangeable.
This expansion brings a series of problems, such as soaring training and inference costs, as well as the need for larger and higher-quality datasets to feed these models. Without such datasets, no matter how large the model, it won't produce satisfactory results.
The problem the authors of this paper aim to solve is:
- Given a fixed computation (FLOPs) budget, how should we choose the optimal combination of "model size" and "dataset size" to minimize the final pre-training loss of the language model?
Kaplan et al.’s Scaling Laws already studied compute allocation. Chinchilla questions the use of intermediate losses from a fixed training horizon and learning-rate schedule: a shorter run can look worse if its decay schedule is not shortened accordingly.
- The Scaling Laws analysis tends to suggest "make the model bigger, but don’t increase the dataset size that much".
Matching the learning-rate schedule to each training horizon leads the authors to recommend scaling model size and training-token count in approximately equal proportions.
A more formal description is that the authors define the optimal solution as follows:
Where:
- is the number of model parameters,
- is the number of training tokens (representing the amount of training data),
- is the fixed computation budget (i.e., how many FLOPs you can spend to train with the given compute power and time),
- is the pre-training loss of the model after training with parameters and training tokens.
The goal of this paper is to optimize the combination of "model size (N)" and "training token count (D)" within a fixed computation budget (C).
Solving the Problem
To reduce the difficulty of reading the following article, please remember the corresponding terms:
- : Fixed computation (FLOPs) budget,
- : Model size (number of parameters),
- : Training data size (number of training tokens),
The authors propose three different methods to estimate the "optimal scaling relationship of and with respect to ."
Approach 1: Fixed Model Size ()
In this approach, the authors first "fix a set of model sizes," and for each model size, they try different training data amounts.
Model sizes range from around 70M parameters to over 10B parameters, with four independent training runs for each size. The training data amounts vary by up to 16 times. During training, the learning rate decays according to a schedule, and for each setting, smoothing and interpolation are applied to obtain continuous loss curves.
With this experimental design, they can intuitively answer: "Given a specified FLOPs budget , which combination of achieves the minimum loss ?"
From a mathematical perspective, the authors aim to solve the following optimization problem:
In this process, they chose 1500 different values distributed on a logarithmic scale (i.e., different FLOPs budgets) and observed the corresponding and .
Through fitting, they found that the optimal parameter configuration approximately follows:
This means that when available compute power increases, the best strategy is to increase both model parameters and training tokens at about the same rate of . This result is a significant departure from previous literature, which mostly suggested that the model size should increase faster than the training data size.
The experimental results are shown in the figure below:
On the left of the figure, each curve corresponds to different model sizes and training data amounts. The x-axis roughly represents the FLOPs used during training, and the y-axis shows the loss of the model during training.
The authors first plot all these training curves, then take the envelope, i.e., the minimum possible loss achieved in each experiment at any given FLOPs budget .
With the envelope, we can see: given a fixed computation budget, is there a better combination that can achieve a lower loss? If so, that would mean we can adjust the ratio of model size and training tokens to get better results.
Next, the middle graph shows the inferred "optimal model parameter count " based on the envelope at different FLOPs . In simple terms, this tells us the optimal model size for a specific FLOPs budget.
The right graph shows how the "optimal number of training tokens " changes with FLOPs. After extracting these points, the authors fit them with a power-law relationship to find scaling relationships like and .
The green dotted line in the figure represents the FLOPs used by the Gopher model (around ) for comparison.
Based on these envelope lines and the inferred power-law relationships, it is evident that the optimal and differ significantly from Gopher's original design. This means that by adjusting the configuration according to the analysis in this paper, one could achieve a lower loss with the same compute budget.
Approach 2: Fixed Compute Budget ()
In this method, the authors first fix a set of FLOPs budgets, and for each fixed , they adjust model size and training token count to ensure that the total FLOPs spent during training exactly equals the specified value.
In simple terms, they "fix the FLOPs consumption," and along this "constant FLOPs line," they examine how different model sizes affect the final loss .
Theoretically, this graph usually forms a "U-shaped curve" (or something like the bottom of a parabola), indicating that there is an optimal model size that minimizes the loss under fixed compute. They fit this curve with a parabolic function to find the position of the minimum.
Through this fitting, the authors obtain a set of data points that can be fitted with a power law:
The experimental results are shown in the figure below:
They calculate and .
Similarly, this suggests that as the compute budget increases, the best strategy is to let model size and training token count grow together at roughly (because 0.5 power means taking the square root).
Finally, they apply this result to estimate the FLOPs budget used in Gopher training (green markers in the figure).
The conclusion is similar to the previous method: current large models may be too big, with too few training tokens. Based on these results, if we reduce the model size and increase the training token count under the same compute budget, we could theoretically achieve a better final loss.
Approach 3: Unified Parametric Model
Since we have a wealth of experimental data from "Approach 1" and "Approach 2," why not directly use an appropriate mathematical model to fit all the final loss data simultaneously and then see what the optimal scaling rule derived from this is?
Assume a parametric functional form as follows:
In this model:
- is the fitted irreducible-loss term, motivated by the entropy of the data distribution; its fitted value is not an exact measurement of natural-language entropy.
- represents a bias that still exists in a finite model size, even when training the model to its maximum capacity. As increases, the model approaches the ideal generation process, but the gap is never completely eliminated.
- represents the "under-training" error due to insufficient training tokens and insufficient training steps. As increases, this error decreases.
The authors then use the L-BFGS optimizer to minimize the Huber loss, ensuring that closely matches the actual observed experimental loss on the logarithmic scale ().
The objective function here is:
Once the parametric model is obtained, the authors can solve the optimization problem under the fixed FLOPs budget to obtain and .
For the dense Transformers studied here, training compute is approximated by . Substituting this constraint gives:
Where:
The estimates are approximately and : near-equal scaling, rather than exactly equal exponents.
Finally, they use this parametric model to plot various iso-loss contours and iso-FLOP slices, and use the blue "efficient frontier" to show where the minimum loss can be achieved with the same compute budget, as shown in the figure below:
When applied to the FLOPs budget used by Gopher, they predict the optimal model size should be about 40B parameters, much smaller than the original 280B in Gopher, while requiring more training tokens.
Discussion
Despite the different implementation details and analytical approaches in the first three methods, the recommendations they provide are very similar:
- As computation resources (FLOPs) increase, the best approach is to expand both model parameters and training data (token count) in almost "equal proportion."
In other words, if your compute power increases, the optimal strategy is and , with both and close to 0.5.
In contrast, Scaling Laws' conclusions (as shown in the table above) suggest that should grow faster () and should grow more slowly (). The analysis in this paper indicates that previous viewpoints may have underestimated the benefits of increasing the training token count, meaning the "importance of data" has been overlooked.
The authors then list the required FLOPs and corresponding training tokens for different model sizes, when situated on the "compute-optimal frontier," in the following table:
Here, “compute-optimal” means choosing the configuration with the lowest loss for a fixed training budget. Further training gains and lifetime serving cost require separate evaluation.
For example, Table 3 projects that bringing a 175B-parameter model, about the size of GPT-3, onto this frontier would require roughly FLOPs and 3.7 trillion tokens, far more training tokens than GPT-3 used.
At Gopher’s budget of about FLOPs, Approach 1 estimates about 67B parameters and 1.5 trillion tokens. Compared with Gopher’s actual 280B parameters and 300B tokens, that is roughly one quarter as many parameters and five times as many training tokens. Approach 3 instead predicts about 40B parameters. These are different fitted estimates, not the configuration of the final trained model.
Chinchilla's Performance
The final Chinchilla model has 70B parameters and is trained on 1.4 trillion tokens, compared with Gopher’s 280B parameters and 300B tokens, at similar training FLOPs. The figure compares language modeling on The Pile subsets; the paper also reports downstream evaluations such as MMLU.
Chinchilla has lower bits-per-byte (bpb) than Gopher on every plotted The Pile subset. This is negative log-likelihood normalized by byte count; lower is better, but it is not the fraction of correctly predicted text. Table A5 shows only two subsets where Chinchilla scores worse than Jurassic-1: dm_mathematics and ubuntu_irc. The authors caution that Chinchilla used more training data, so overlap between training and test data could affect these language-modeling comparisons.
Conclusion
The scaling fits use over 400 smaller training runs. At the large end, however, the direct validation consists of Chinchilla versus Gopher, without further intermediate-scale validation runs.
Power-law extrapolation has limits: curvature at high compute may imply that optimal model size is still overestimated. The scaling runs use less than one epoch, so the result does not directly establish behavior under heavy data repetition. Final Chinchilla training uses weighted sampling; MassiveWeb and Wikipedia each exceed one effective epoch. Not every part of the study sees every datum only once.
Nevertheless, the results of Chinchilla relative to Gopher nicely confirm their theory: with the same training costs, using smaller models trained on more data indeed outperforms models that are purely scaled up in size. Researchers should place more emphasis on expanding and improving the quality of datasets rather than focusing solely on model size.
Finally, the authors speculate that this "model size vs. training data" optimization relationship is not limited to language models, but also applies to the training of models in other domains. By estimating the optimal model size and training schedule in advance, much resource wastage can be avoided, which is a valuable insight for the future research community.