Back to paper library
arXiv 2020Language & TransformersPaper notes

Scaling Laws

Scaling Laws for Neural Language Models

Original paper

Scaling Laws for Neural Language Models

Paper authors

Jared Kaplan · Sam McCandlish · Tom Henighan …

Show all 10 authors

Jared Kaplan · Sam McCandlish · Tom Henighan · Tom B. Brown · Benjamin Chess · Rewon Child · Scott Gray · Alec Radford · Jeffrey Wu · Dario Amodei

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2001.08361
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformersscalinglaws2020,
  title = {{Scaling Laws for Neural Language Models}},
  author = {Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei},
  year = {2020},
  eprint = {2001.08361},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.08361}
}
On this page

OpenAI has been pushing the boundaries with the Transformer decoder architecture.

Problem Definition

From previous articles, you might have noticed OpenAI's ambition: they aim to build a "super-large" language model.

Thus, this study addresses a fundamental question:

  • How exactly should one scale up a model?

This question is the core challenge this paper aims to solve.

Solution

In this paper, the following hyperparameters are used to parameterize the Transformer architecture:

  • The dimension of the residual stream (model width)
  • The dimension of the feed-forward layer
  • The dimension of the attention output
  • The number of attention heads per layer
  • The number of layers: The depth of the Transformer
  • The number of tokens: The context length, typically set to 1024

The paper measures language-model performance by autoregressive cross-entropy loss on the WebText2 test set. Below, model size NN counts parameters excluding vocabulary and positional embeddings.

Training Process

Unless otherwise specified, the models are trained using the Adam optimizer for 2.5×1052.5\times10^{5} steps, with a batch size of 512 sequences, each containing 1024 tokens. Due to memory limitations, models with more than 1 billion parameters use Adafactor for training.

The authors tested several learning rates and schedules. In these settings, results near convergence were relatively insensitive to the choice of schedule. Unless stated otherwise, runs use a 3000-step linear warm-up followed by cosine decay to zero.

Dataset

The models are trained on an "extended version of the WebText dataset."

  • The original WebText dataset uses outbound Reddit links through December 2017 with at least three karma points; that threshold is not a count of individually checked readers.
  • WebText2 adds outbound links from January through October 2018 under the same minimum three-karma threshold.
  • The text from these links was extracted using the Newspaper3k Python library, resulting in a dataset containing 20.3 million documents, totaling 96GB of text and 1.62×10101.62\times10^{10} words.
  • Byte-pair encoding (BPE) was applied for tokenization, yielding 2.29×10102.29\times10^{10} tokens, with 6.6×1086.6\times10^{8} tokens reserved for the test set.
  • The models were evaluated on Books Corpus, Common Crawl, English Wikipedia, and other publicly available online book samples.

Variation Factors

To comprehensively study the scaling characteristics of language models, this paper conducted extensive model training with variations in the following aspects:

  • Model size: Models ranging from 768 to 1.5 billion non-embedding parameters
  • Dataset size: Datasets ranging from 22 million to 23 billion tokens
  • Model shape: Including depth, width, number of attention heads, and feed-forward layer dimensions
  • Context length: Most training uses a context length of 1024, but shorter contexts were also tested
  • Batch size: Most training uses 2192^{19} tokens per batch—that is, 512 sequences of 1024 tokens—but batch sizes were varied to measure the critical batch size

These experimental designs aim to systematically investigate the performance of models under different conditions and derive relevant scaling laws.

Discussion

Size Matters Most

Within the study’s experimental range, language-model cross-entropy loss depends strongly on scale and less on moderate changes in depth-to-width ratio. The three scale variables are:

  1. N: Number of non-embedding parameters
  2. D: Dataset size
  3. C: Training compute used

Experiments indicate that within a reasonable range, performance dependency on other architectural hyperparameters (such as depth and width) is minimal.

Power Laws

When the other two factors are not bottlenecks, test loss follows an approximate power law with each of NN, DD, and compute. The plots cover different horizontal ranges. In the left plot, compute is the batch-efficiency-adjusted minimum CminC_{min}, rather than raw compute measured for every run.

  • Left chart: As CminC_{min} increases, the lower envelope of test loss falls; pale-blue lines show individual training runs.
  • Middle chart: With model shape fixed, increasing the number of dataset tokens lowers early-stopped test loss.
  • Right chart: When trained toward convergence on enough data, models with more non-embedding parameters have lower test loss.

The Universality of Overfitting

Experiments indicate that when NN and DD grow together, early-stopped test loss predictably falls. If one is held fixed while the other grows, returns diminish. In this fit, the extra loss from the data bottleneck depends mainly on N0.74/DN^{0.74}/D: an eightfold increase in parameters calls for roughly five times as much data to keep the bottleneck at a similar level.

Test Performance vs. Downstream Performance

When the models are evaluated on text from a distribution different from the training data, their loss tracks loss on the training validation set closely, with a roughly constant offset.

In other words, transfer to another text distribution incurs a roughly constant loss penalty, but otherwise improves along with in-distribution performance. This is about language-model loss, not a fixed accuracy gain on every downstream task.

Sample Efficiency

This imported image combines paper Figure 2 on top and Figure 4 below. The upper-left plot uses tokens processed as its horizontal axis, and the upper-right uses compute. The lower-left compares dataset sizes, while the lower-right compares estimated minimum training steps. For the same test loss, a larger model usually needs fewer data tokens or steps, even though each step costs more compute.

Convergence is Inefficient

Figure 3 illustrates a billionfold increase in compute allocated among model size, batch size, and serial steps according to the authors' fitted laws. It is a scenario, not a measured training curve.

For a fixed compute budget under the paper's WebText2 fit, and with no separate limit on model size or available data, choosing a larger model and stopping before convergence can yield lower loss than training a smaller model to convergence. The fit predicts rapid growth in the compute-efficient model size and much slower growth in data processed. If data is limited, repeatedly reused, or compute goes far beyond the measured range, the allocation in the figure cannot simply be carried over. Later Chinchilla also finds a different parameter/data allocation.

Optimal Batch Size

The optimal batch size for training language models is not fixed but related to the loss function and can be determined by measuring the gradient noise scale.

For the largest model (about 1.5B non-embedding parameters), the estimated critical batch size near convergence is roughly 1–2 million tokens. It changes with the target loss and marks an approximate trade-off between training steps and compute efficiency, not a fixed setting for every phase.

Conclusion

This paper conducted extensive experiments and summarized the training methods for language models.

To us, these conclusions are represented by large sums of money, and we gratefully accept these experimental results.