Back to paper library
arXiv 2019Language & TransformersPaper notes

RoBERTa

A Guide to Training BERT

Original paper

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Paper authors

Yinhan Liu · Myle Ott · Naman Goyal …

Show all 10 authors

Yinhan Liu · Myle Ott · Naman Goyal · Jingfei Du · Mandar Joshi · Danqi Chen · Omer Levy · Mike Lewis · Luke Zettlemoyer · Veselin Stoyanov

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:1907.11692
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformersroberta2019,
  title = {{RoBERTa: A Robustly Optimized BERT Pretraining Approach}},
  author = {Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov},
  year = {2019},
  eprint = {1907.11692},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.11692}
}
On this page

BERT’s success inspired a wave of subsequent research, and this paper revisits its design with a focus on optimization. Through a series of experiments, the authors discovered several areas where BERT could be improved significantly.

Defining the Problem

Let’s begin by reviewing the key aspects of BERT's design:

BERT's pretraining involves two main objectives: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).

BERT first undergoes pretraining on large-scale unlabelled text, followed by fine-tuning on labelled data.

Built on a Transformer architecture, BERT consists of L layers of Transformers, each with A self-attention heads, and hidden layers of dimension H.

BERT’s pretraining objectives consist of:

  • Masked Language Modeling (MLM):

    • Randomly masking 15% of the tokens in the input sequence, where the objective is to predict the masked tokens. Of the selected tokens, 80% are replaced with a [MASK] token, 10% remain unchanged, and 10% are replaced with random tokens.
  • Next Sentence Prediction (NSP):

    • NSP is a binary classification task that predicts whether two segments of text are consecutive. Positive samples come from contiguous sentences, while negative samples are from different documents. Each type of sample represents 50% of the training data.

BERT uses the Adam optimizer with the following hyperparameters:

  • β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, and L2 regularization with a weight decay of 0.01.
  • The learning rate gradually increases to a peak of 1e-4 over the first 10,000 steps and then decays linearly.
  • Use 0.1 dropout on hidden states and attention probabilities, with GELU in the FFN. GELU is not applied directly to all attention weights.
  • Pretraining continues for 1,000,000 updates, with a batch size of 256 sequences, each with a maximum sequence length of 512 tokens.

BERT was trained on a combination of the BOOKCORPUS and English WIKIPEDIA datasets, which together amount to 16GB of uncompressed text.


The authors looked back at all these choices and thought: there is still a lot to improve. So they set out to change BERT's training recipe.

Solving the Problem

Optimization of Implementation Details

Most hyperparameters follow the original BERT optimization settings, except for adjustments in the peak learning rate and the number of warm-up steps, which vary depending on the specific settings.

The authors found that training is highly sensitive to the epsilon value in the Adam optimizer, with adjustments potentially improving performance or stability. When using larger batch sizes, setting β2=0.98\beta_2 = 0.98 also enhances stability.

Unlike the earlier BERT training setup, the authors did not randomly insert short sequences or reduce sequence length for the first 90% of updates. They used full-length sequences, up to 512 tokens, throughout pre-training.

Training was conducted using mixed-precision floating-point computation on DGX-1 machines, each equipped with 8 × 32GB Nvidia V100 GPUs.

Optimizing the Training Data

Pretraining BERT-like models requires a large corpus of text data. Prior research showed that increasing the dataset size generally improves performance in downstream tasks.

Though many studies used larger and more diverse datasets than the original BERT, not all of these additional datasets are publicly available. In their experiments, the authors collected as much data as possible to ensure high data quality and quantity.

In the end, the authors utilized five different English corpora of varying sizes and domains, totaling over 160GB of uncompressed text. The first list item groups two of the five corpora:

  1. BOOKCORPUS and WIKIPEDIA: The same datasets used in the original BERT training (16GB).

  2. CC-NEWS: English news articles from the CommonCrawl News dataset, spanning from September 2016 to February 2019, containing 63 million articles (after filtering, 76GB).

  3. OPENWEBTEXT: An open-source recreation of the WebText dataset described by Radford et al. (2019), containing web pages linked from Reddit posts with at least three upvotes (38GB).

  4. STORIES: A dataset introduced by Trinh and Le (2018), consisting of text from CommonCrawl that matches the Winograd schema story style (31GB).

Discussion

In the paper, the authors break down the key areas of improvement into several topics:

Static Masking vs. Dynamic Masking

Paper Table 1; source: arXiv:1907.11692v1

Table: Paper Table 1. The static/dynamic rows belong to the authors' BERT-Base reimplementation; the reference row is from another reported result.

The original BERT implementation used static masking, where the masks were applied once during preprocessing and reused for every epoch. To avoid training on the same masked patterns, the data was duplicated 10 times, allowing each sequence to be masked in 10 different ways over 40 epochs, with each masked version being seen four times.

Dynamic masking, on the other hand, generates new masking patterns each time the sequence is fed into the model, making it particularly useful for longer training steps or when larger datasets are used for pretraining.

Results are mixed. Relative to static masking, dynamic masking changes SQuAD 2.0 from 78.3 to 78.7 F1 and SST-2 from 92.5 to 92.9%, but MNLI-m from 84.3 to 84.0%. The authors treat the methods as broadly comparable and use dynamic masking for its fresh samples at each read, not because it wins every task.

Next Sentence Prediction (NSP)

Paper Table 2; source: arXiv:1907.11692v1

Table: Paper Table 2. The no-NSP configurations also change how sentences are packed, so these rows do not isolate the NSP loss alone.

Original NSP uses adjacent segments from the same document as positives and randomly drawn segments from another document as negatives. Its target follows this sampling scheme, not a general test of whether any two segments share a source.

Some research had questioned the necessity of NSP, suggesting that removing it might not negatively affect performance.

To evaluate this, the authors compared several input formats:

  • SEGMENT-PAIR+NSP: The same as original BERT, retaining the NSP loss, with input consisting of two segments.
  • SENTENCE-PAIR+NSP: Input consists of two natural sentences, retaining NSP loss. These inputs are much shorter than 512 tokens, so the batch size is increased to keep the total token count similar.
  • FULL-SENTENCES: Input packs consecutive full sentences, NSP loss is removed, and sequences may cross document boundaries, with an extra separator when they do.
  • DOC-SENTENCES: Similar to FULL-SENTENCES, but sequences do not cross document boundaries, and NSP loss is removed. Near a document's end, the batch size is adjusted to keep a similar total token count.

Comparing SEGMENT-PAIR+NSP with DOC-SENTENCES, SQuAD 2.0 rises from 78.7 to 79.7 F1, MNLI-m from 84.0 to 84.7%, and RACE from 64.2 to 65.6%, while SST-2 falls from 92.9 to 92.7%. The no-NSP configuration matches or improves most tasks here, but sentence packing changes too; Table 2 cannot assign the entire difference to NSP removal.

DOC-SENTENCES does slightly better than FULL-SENTENCES, but the authors use FULL-SENTENCES later because its batch size is easier to keep fixed for comparisons.

Large Batch Training

Paper Table 3; source: arXiv:1907.11692v1

Table: Paper Table 3. Step counts and learning rates also change with batch size; 2K, not 8K, gives the best perplexity and MNLI-m in this table.

Batch size must be compared together with learning rate and total data exposure. The reference uses BERT-Base with batch 256 and one million updates; the comparison adjusts batch size and steps to preserve the total number of examples seen.

At batch sizes 256/2K/8K, perplexity is 3.99/3.68/3.77, MNLI-m is 84.7/85.2/84.6%, and SST-2 is 92.7/92.9/92.8%. The 2K row is best on all three, so gains are not monotonic. Larger batches facilitate distributed data parallelism, and the authors choose 8K for later runs. Since learning rate and update count change as well, this is not a single-variable batch ablation.

Tokenization

Byte-Pair Encoding (BPE) can handle large vocabularies in natural language. BPE treats subword units as the basic unit of text.

Original BERT uses about 30K WordPiece entries after heuristic preprocessing; its tokens are not all individual characters. RoBERTa switches to about 50K byte-level BPE entries to reduce reliance on language-specific preprocessing.

Initial experiments showed little difference in performance between the two, with byte-level BPE performing slightly worse in some tasks. However, due to its broader applicability, the authors used byte-level BPE in subsequent experiments.

Training Data Size

Paper Table 4 with legacy red annotations; source: arXiv:1907.11692v1

Table: The imported crop of paper Table 4 includes red boxes and an English arrow added in the earlier note; they are not original paper annotations. Its first RoBERTa row uses 16GB, while the published BERT-Large baseline uses 13GB, not identical data volume.

The model shares BERT-Large’s layer and hidden configuration (L = 24, H = 1024, A = 16). The larger vocabulary brings RoBERTa-Large to about 355M parameters, compared with original BERT-Large’s 340M. The first run uses BOOKCORPUS and WIKIPEDIA for 100K steps.

Using 1024 V100 GPUs, training took roughly one day.

The first RoBERTa row uses 16GB, an 8K batch and 100K steps, giving 87.3 F1 on SQuAD 2.0 and 89.0% on MNLI-m. The published BERT-Large row uses about 13GB, a 256 batch and 1M steps, giving 81.8/86.6. Training choices matter, but this is not a single-variable comparison. RoBERTa's own rows are clearer: raising the corpus from 16GB to 160GB at 8K/100K steps changes SQuAD 2.0 87.3→87.7 and MNLI-m 89.0→89.3; extending the 160GB run to 500K steps brings them to 89.4/90.2. These rows accumulate changes.

Conclusion

Key areas of improvement highlighted in the paper:

  1. Train the model for longer.
  2. Use larger batches and train on more data.
  3. Remove the "NSP" objective.
  4. Train with longer input sequences.
  5. Use dynamic masking on the training data.

RoBERTa reached then-best results on GLUE, RACE and SQuAD, without multi-task fine-tuning for GLUE or extra training data for SQuAD. Its GLUE test submission did ensemble single-task models, and some tasks used specific fine-tuning steps. The results show how much these overlooked training choices can matter.

The same idea carries over to other architectures: a good training recipe can bring out more of a model's performance. When making a comparison, we still have to keep the corpus, training compute and fine-tuning procedure in view.