ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Publication
- ICLR 2020Conference
- First submitted to arXiv
- arXiv:1909.11942
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformersalbert2019,
title = {{ALBERT: A Lite BERT for Self-supervised Learning of Language Representations}},
author = {Zhenzhong Lan and Mingda Chen and Sebastian Goodman and Kevin Gimpel and Piyush Sharma and Radu Soricut},
year = {2019},
eprint = {1909.11942},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1909.11942}
}
After BERT, there's still more BERT.
The Problem
A BERT-Large model has around 340M parameters (334M in this paper's Table 2), and training it takes time. With BERT as a point of comparison, subsequent models kept getting bigger.
0.1B ➔ 0.3B ➔ 0.5B ➔ 1.0B ➔ 1.5B ➔ ???
The authors' response is: no, we cannot just keep making them bigger. Soon we will not be able to train them!
- We need a smaller BERT!
Here “smaller” means fewer trainable parameters. The authors still want wider hidden layers and more representation capacity.
The Solution
ALBERT keeps BERT's Transformer encoder backbone. Two changes reduce its parameter count, and a third changes the pretraining task for inter-sentence relations.
Factorized Embedding Parameterization
In BERT and later models such as XLNet and RoBERTa, the vocabulary-embedding dimension is tied to hidden size : . The paper calls these “WordPiece” embeddings loosely; RoBERTa uses byte-level BPE, and ALBERT's experiments use a 30,000-item SentencePiece vocabulary. The tokenizers are not identical.
This has two main drawbacks:
-
From a modeling perspective: WordPiece embeddings primarily learn context-independent representations, whereas hidden layer embeddings focus on context-sensitive representations. By decoupling and , the model can allocate parameters more effectively. Hidden layers should have larger capacity ().
-
From a practical perspective: Vocabulary size is usually large. If , widening the hidden layers also enlarges the vocabulary matrix. Those weights occupy memory and add parameter communication in distributed training; only the vocabulary rows used by a batch are updated in that batch.
ALBERT therefore factorizes the vocabulary embedding into two matrices:
- A matrix maps tokens into a lower-dimensional embedding space.
- An matrix then projects them into the Transformer's hidden space.
This factorization reduces the number of embedding parameters from to .
Factorization saves substantially when is much smaller than and the vocabulary is large; the exact parameter-saving condition is .
Cross-layer Parameter Sharing
Several prior works have focused on improving model parameter efficiency:
-
Dehghani et al. (2018) with Universal Transformer (UT):
UT also reuses Transformer parameters. The ALBERT paper cites its language-modeling and subject-verb-agreement results to show that sharing weights predates ALBERT. UT's recurrent computation differs from the BERT-style pretraining objective here.
-
Bai et al. (2019) with Deep Equilibrium Models (DEQ):
DEQ studies a fixed point of a repeated transformation, where input and output representations agree at equilibrium. Sharing parameters alone does not make ALBERT a fixed-point model.
ALBERT takes inspiration from these approaches, implementing cross-layer parameter sharing in several ways:
- Sharing only the parameters of the feed-forward network (FFN) across layers.
- Sharing only the attention mechanism parameters.
ALBERT shares Transformer-layer parameters, including attention and FFNs, by default. Reusing the same weights still requires a computation at every layer: it saves parameter storage and updates, not 24 layers of work. The figure measures L2 distance and cosine angle between each layer's input and output representations, not changes in the embedding weights.
Weight sharing and DEQ fixed-point solving are separate design choices; the role of oscillation in performance requires its own analysis.
Sentence-Order Prediction Loss (SOP)
Alongside MLM, BERT used next-sentence prediction (NSP): positive pairs are adjacent segments from one document, whereas negative pairs come from different documents. Later studies found its downstream benefits unreliable. The ALBERT authors suspect NSP is too easy beside MLM: it mixes topic prediction with coherence prediction, and the easier topic clue overlaps with what MLM already learns.
The authors emphasize that modeling sentence-level coherence is important for language understanding and propose the Sentence-Order Prediction (SOP) loss:
- Positive examples are the same as BERT, two consecutive segments from the same document.
- Negative examples are also two consecutive segments from the same document, but with their order swapped.
Both labels use the same pair from the same document, making topical clues less revealing. The model must judge order and coherence. That is the authors' motivation; the SOP and downstream ablations below test whether it helps.
Discussion
BERT vs. ALBERT: Overall Comparison
This is a dev-set comparison after 125K pretraining steps on BookCorpus and Wikipedia. The BERT rows are the paper authors' runs under the same experimental setup; they should not be mixed with the larger-corpus leaderboard results below.
ALBERT-large has 18M parameters versus BERT-large's 334M, about 18 times fewer, yet its Avg is 82.4 versus 85.2. Parameter savings at this size do not preserve every score. After widening the model, ALBERT-xxlarge has 235M parameters, about 70% of BERT-large, and scores 88.7 versus 85.2 on Avg and 82.3 versus 73.9 on RACE. Its SQuAD 1.1, SQuAD 2.0, MNLI and SST-2 scores are higher by 1.9, 3.1, 1.4 and 2.2 points. Hidden width also changed, so the entire gain cannot be attributed to one component.
With the same TPU count, ALBERT-large has 1.7 times BERT-large's training data throughput, while xxlarge falls to 0.3 times. This measures training throughput, not single-request inference latency. Parameter count and FLOPs measure different costs.
Embedding Parameter Factorization
The table above shows the impact of varying the embedding size on ALBERT-base.
Without sharing, reaches Avg 82.3 versus 81.7 for , at 108M versus 89M parameters. With full sharing, scores 80.1 versus 79.8 for , at 12M versus 31M parameters. The best depends on the sharing setup.
The subsequent ALBERT configurations therefore use as they widen the hidden layers.
Cross-layer Parameter Sharing
The table above presents the results of experiments with different cross-layer parameter sharing strategies:
- Using ALBERT-base configuration ( and ).
Against the respective not-shared baselines, all-shared changes Avg from 81.6→80.1 at , a 1.5-point loss, and from 82.3→79.8 at , a 2.5-point loss. The large parameter reduction has a measurable score tradeoff.
At , sharing only attention reaches Avg 81.7 versus 81.6 with no sharing, while sharing only the FFN scores 80.2. At , these scores are 81.6 and 79.5 versus 82.3 without sharing. The larger loss is associated with FFN sharing in these settings.
While dividing layers into groups and sharing parameters within each group could be considered, the experiments show that the smaller the group, the better the performance, but the more parameters are required.
Therefore, the authors ultimately chose the full parameter-sharing strategy as the default.
The Effectiveness of Sentence-Order Prediction
The authors compared three types of cross-sentence loss conditions:
- None (XLNet and RoBERTa style).
- NSP (BERT style).
- SOP (ALBERT style).
The NSP-trained model scores 90.5% on NSP but only 52.0% on SOP, close to the no-sentence-task model's 53.3%. The SOP-trained model scores 78.9% on NSP and 86.5% on SOP. This supports the authors' topic-shortcut explanation.
Relative to the NSP condition, SOP changes SQuAD 1.1 F1 88.4→89.3, SQuAD 2.0 F1 77.2→80.0, and RACE 62.3→64.0. SST-2 instead falls 91.1→90.3, so the gain is not universal. Table 5's Avg rises 79.2→80.1.
State-of-the-art Results on NLU Tasks
ALBERT on GLUE: single-model development results above, ensemble test results below.
Source: paper Table 9.
ALBERT on SQuAD and RACE: single-model and ensemble results.
Source: paper Table 10.
Finally, the authors use expanded pretraining data, MLM plus SOP, no dropout, and an xxlarge configuration to report both single-model and ensemble results. These are not the 125K-step baseline conditions of Table 2.
- Single-model:
- Median results of five runs are reported on the development set.
- Table 9 includes ALBERT at 1M and 1.5M steps. In Table 10, the 1.5M-step single model gets 86.5% RACE test accuracy and 90.9 SQuAD 2.0 test F1.
- Ensemble models:
- Checkpoints are selected based on development set performance, ranging from 6 to 17 depending on the task.
- On GLUE and RACE, predictions from fine-tuned 12- and 24-layer configurations at different pretraining steps are averaged. SQuAD ensembles average answer-span and “unanswerable” decision scores.
The following are ensemble test results, not single-model scores:
- GLUE: ALBERT achieves a score of 89.4.
- SQuAD 2.0: F1 score reaches 92.2.
- RACE: Accuracy reaches 89.4.
The 89.4% RACE ensemble result is compared below with single-model BERT, XLNet and RoBERTa entries; DCMN+ is the other ensemble. That difference in setup matters:
- 17.4 percentage points over BERT at 72.0%.
- 7.6 points over XLNet at 81.8%.
- 6.2 points over RoBERTa at 83.2%.
- 5.3 points over the DCMN+ ensemble at 84.1%.
ALBERT's single-model RACE test accuracy is 86.5%, so it should not be mistaken for the ensemble's 89.4%.
Conclusion
ALBERT improves parameter efficiency by separating vocabulary-embedding width from hidden width and reusing Transformer weights across layers. It also introduces a more targeted SOP loss for learning relations between text segments. These ideas have far-reaching implications for later large language models, though Table 5's SST-2 result is an exception to the SOP gains.
If you are building a language model with limited resources, ALBERT may be worth considering! Its parameter savings help with memory; the xxlarge variant still takes more computation per training step than BERT-Large, so the choice depends on which resource is scarce.