Back to paper library
arXiv 2020NeurIPS 2020Language & TransformersPaper notes

GPT-3

Ninety-Six Layer Decoder

Original paper

Language Models are Few-Shot Learners

Paper authors

Tom B. Brown · Benjamin Mann · Nick Ryder …

Show all 31 authors

Tom B. Brown · Benjamin Mann · Nick Ryder · Melanie Subbiah · Jared Kaplan · Prafulla Dhariwal · Arvind Neelakantan · Pranav Shyam · Girish Sastry · Amanda Askell · Sandhini Agarwal · Ariel Herbert-Voss · Gretchen Krueger · Tom Henighan · Rewon Child · Aditya Ramesh · Daniel M. Ziegler · Jeffrey Wu · Clemens Winter · Christopher Hesse · Mark Chen · Eric Sigler · Mateusz Litwin · Scott Gray · Benjamin Chess · Jack Clark · Christopher Berner · Sam McCandlish · Alec Radford · Ilya Sutskever · Dario Amodei

Publication
NeurIPS 2020Conference
First submitted to arXiv
arXiv:2005.14165
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformersgpt32020,
  title = {{Language Models are Few-Shot Learners}},
  author = {Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei},
  year = {2020},
  eprint = {2005.14165},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.14165}
}
On this page

The largest GPT-2 variant had 48 decoder layers. Apparently OpenAI wanted more again: the largest GPT-3 went up to 96 layers and 175B parameters. That is 96 layers total, not another 96 stacked on top of the previous 48. The study trained smaller variants too, so it could compare model scale with in-context learning.

Problem Definition

Recent work has demonstrated that by pre-training on a large corpus of text and then fine-tuning on specific tasks, significant progress can be made on many NLP tasks and benchmarks. Ideally, an NLP system should be able to quickly adapt and handle a variety of language tasks with minimal instruction or examples, much like a human. However, current research shows a significant gap between this ideal and reality, presenting several challenges:

Diverse Language Task Requirements

Current language models face the need to adapt to a wide range of language tasks, from grammatical corrections to abstract concept generation. Each new task typically requires a large, specific labeled dataset, limiting the model's general applicability.

Dependence on Large Supervised Datasets

Collecting and annotating large datasets for many language tasks is expensive and time-consuming. Each new task necessitates repeating this data collection process, increasing development costs and extending deployment times.

Over-Specialization and Generalization Issues

Current models, when fine-tuned on specific tasks, may become overly specialized, resulting in poor generalization to data outside the training distribution. Training can also introduce spurious correlations that mislead the model, affecting its long-term and broad applicability.

Human Learning Efficiency vs. Model Efficiency

Humans can quickly adapt to new tasks with minimal examples or direct instructions, whereas current models rely heavily on large amounts of labeled data, making them inefficient in comparison.

Solution

In GPT-3, the basic pre-training method, including the model, data, and training, follows a similar approach to GPT-2 but with simple scaling up of model size, dataset size and diversity, and training length. The use of in-context learning also aligns with GPT-2, but this work systematically explores different settings of in-context learning, mainly differentiated as:

  1. Fine-tuning

    • This is the most common approach, involving updating the pre-trained model's weights by training on a task-specific supervised dataset, typically using thousands to hundreds of thousands of labeled examples. The main advantage of fine-tuning is achieving state-of-the-art performance on many benchmarks.
    • The main drawbacks are the need for a new large dataset for each task, potential poor generalization outside the training distribution, and the possibility of exploiting spurious features in the training data, leading to unfair comparisons with human performance.
    • In this work, the authors did not fine-tune GPT-3, as their focus was on task-agnostic performance. However, GPT-3 can be fine-tuned in principle, which is a promising direction for future work.
  1. Few-shot

    • Few-shot means providing the model with a few task demonstrations as conditions during inference without allowing weight updates.
  • The main advantage is significantly reducing the need for task-specific data and reducing the potential for learning overly narrow distributions from large, narrow fine-tuning datasets.
  • The main drawback is that few-shot results in this study often lagged the strongest task-specific fine-tuned models. It still needs some task examples; the outcome depends on the model, task and demonstrations.
  1. One-shot

    • One-shot is similar to few-shot but only allows one demonstration and requires a natural language description of the task. This method closely resembles how humans communicate some tasks. For example, when generating datasets for human workers (e.g., Mechanical Turk), a task demonstration is typically provided. Without an example, conveying the task's nature or format can be challenging.
  2. Zero-shot

    • Zero-shot supplies the task description or format without any demonstration. It is the most convenient setting and avoids choosing examples, but the lack of examples can make the intended format hard even for a person to understand.

Model Architecture

In this work, the authors used the same model and architecture as GPT-2, including modified initialization, pre-normalization, and reversible tokenization.

The layers alternate dense and locally banded sparse attention, rather than using one sparse pattern throughout. For background, see:

To study the dependence of machine learning performance on model size, the authors trained eight different model sizes, ranging from 125 million to 175 billion parameters across three orders of magnitude, with the largest model named GPT-3.

The table above shows that all models used a context window of 2048 tokens. The models were partitioned across GPUs along the depth and width dimensions to minimize inter-node data transfer. The exact architectural parameters of each model were chosen based on computational efficiency and load balancing across the GPU layout.

Dataset Setup

Common Crawl is a major component of a mix that also includes WebText2, Books1, Books2 and Wikipedia. Raw crawl volume, cleaned token count and actual sampled training tokens are different quantities.

They also found that unfiltered or lightly filtered versions of Common Crawl typically had lower quality than carefully curated datasets.

To improve the dataset's average quality, they took three steps:

  1. Data Filtering and Quality Control: Filtering the Common Crawl data to select data similar to a range of high-quality reference corpora.
  2. Deduplication: Performing fuzzy deduplication at the document level, both within and across datasets, to prevent data redundancy and maintain the integrity of validation sets.
  3. Enhancing Data Diversity: Adding known high-quality reference corpora to the training mix to enhance and diversify the Common Crawl data.

Using this approach, they collected 41 monthly Common Crawl snapshots from 2016 to 2019, containing 45TB of compressed plaintext before filtering and 570GB after filtering, equivalent to approximately 400 billion byte-pair encoded tokens.

Additionally, during training, sampling was not performed by dataset size but more frequently from higher-quality datasets. For example, Common Crawl and Books2 datasets were sampled less than once, while other datasets were sampled 2-3 times. This approach essentially traded a small amount of overfitting for higher-quality training data.

Discussion

The authors tested GPT-3's performance on language modeling and related tasks, including word prediction and sentence or paragraph completion.

LAMBADA

The LAMBADA dataset requires the model to predict the last word of a sentence, necessitating reading the entire context. This benchmark particularly challenges the model's ability to capture long-range dependencies.

  • Recent research indicated diminishing returns from scaling language models on LAMBADA. Doubling model size between two earlier leading results had raised accuracy by only 1.5 percentage points, leading those researchers to question whether expanding hardware and data was a viable path.
  • OpenAI’s answer was, in effect, that the scale had not been large enough: GPT-3 reaches 76.2% zero-shot accuracy, 8.2 percentage points above the previous best (blue line).
  • In the few-shot setting, GPT-3 reaches 86.4%, 18.4 percentage points above that previous best (orange line).

In these LAMBADA experiments, few-shot falls nearly 20 percentage points below zero-shot for the smallest model, but improves sharply by the 2.7B model and helps the largest model. One-shot also trails zero-shot here. This result depends on this task and prompt format: 2.7B is not a universal threshold for few-shot ability.

Closed Book Question Answering

This task measures GPT-3's ability to answer broad factual knowledge questions, typically answered by information retrieval systems locating relevant texts and generating answers. This task is known as the "open-book" setting, where the system can search and use relevant texts to answer questions. Another setting is the "closed-book," where the system directly answers questions without external information.

The authors test GPT-3 on Natural Questions, WebQuestions and TriviaQA. The figure above shows TriviaQA: zero-shot reaches 64.3%, one-shot 68.0%, and few-shot 71.2%. Zero-shot exceeds fine-tuned T5-11B (50.1%) by 14.2 percentage points and T5-11B+SSM (60.5%) by 3.8 points. One-shot matches the 68.0% RAG baseline, which uses retrieval; few-shot exceeds it by 3.2 points. These TriviaQA scores are not averages across the three datasets, and Natural Questions and WebQuestions have different comparisons.

Across these three QA datasets, performance changes relatively smoothly with model size. The authors suggest this may reflect more “knowledge” stored in the model parameters; it is their interpretation of the trend, not a direct measurement of knowledge.

Conclusion

The paper is a 75-page PDF with far more charts than the ones we have covered. The gains in examples such as LAMBADA and TriviaQA help explain why so much later work explored prompt engineering: finding examples and formats that bring out a model’s abilities without changing its weights. Few-shot helps on many tasks, although the paper does not show that every task needs it or that it always beats zero-shot.

Despite significant advancements in text generation and natural language processing, GPT-3 still faces issues such as semantic repetition, loss of coherence, and logical errors. It also shows specific weaknesses in handling commonsense physics and in-context learning behaviors, reflecting limitations in some specific language tasks.

Additionally, concerning bias and fairness, GPT-3 and similar large language models inherently absorb and reflect biases present in their training data, sourced from vast amounts of internet text. These biases manifest in several ways:

1. Gender Bias

GPT-3 demonstrates notable biases when handling gender-related content.

For example, in generated texts about professions, the model often associates males with certain occupations (e.g., tech, engineering) and females with others (e.g., nursing, teaching).

Such biases may reinforce societal stereotypes about gender roles.

2. Racial Bias

Regarding race, language models might generate texts that include or imply certain racial group characteristics, potentially leading to negative descriptions of some groups.

For example, when prompted about individuals of a particular race, the generated text might disproportionately reflect negative or stereotypical traits.

3. Religious Bias

GPT-3 may also exhibit biases in handling religion-related queries, such as using overly generalized or biased descriptions when discussing characteristics of different religious followers.


These challenges and their solutions underscore the importance of balancing technological progress with ethical responsibility in developing large language models, highlighting the need for ongoing research and improvement to mitigate unfairness.

GPT-3 and similar language models show significant potential in enhancing text generation and natural language processing capabilities but face several challenges. Future research must balance technological advancements with ethics, fairness, and efficiency to ensure these advanced technologies serve humanity while promoting fair and sustainable development.