Longformer: The Long-Document Transformer
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2004.05150
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformerslongformer2020,
title = {{Longformer: The Long-Document Transformer}},
author = {Iz Beltagy and Matthew E. Peters and Arman Cohan},
year = {2020},
eprint = {2004.05150},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2004.05150}
}
The computational complexity of the Transformer’s self-attention mechanism is .
When an algorithm has this kind of complexity, it feels as if there is plenty of room to optimize it.
Wait, does this introduction sound familiar?
Problem Definition
ltr means left-to-right; char-LM, other tasks and pretrain are separate columns.To address the computational complexity issue of the original Transformer, researchers have continuously sought optimization methods.
One approach follows Transformer-XL, processing long text in segments and passing memory from earlier segments to later ones.
The limitation concerns Transformer-XL’s directional segment memory and its adaptation to BERT-style bidirectional context. Sliding windows themselves can be bidirectional, as in Longformer.
Another approach is sparse attention mechanisms, such as the Sparse Transformer proposed a year before this paper.
Table 1 also shows that Blockwise already combined pre-training with QA, while BP-Transformer evaluated machine translation. The gap is a joint evaluation of long-document language modeling, pre-training followed by fine-tuning, and multiple downstream tasks, not an absence of all earlier pre-training.
The motivation for this paper becomes clear:
- Using “sparse attention mechanisms” for long contexts is a promising approach.
- However, the design of the attention pattern needs optimization.
- Moreover, pre-training frameworks should be introduced, and generalization across more downstream tasks should be tested.
Solution
Model Architecture
This paper is relatively straightforward to understand because the image above summarizes the core ideas.
Full attention is the reference in the figure; the other three panels show sparse connection patterns for long documents:
- The figure (a) shows the original attention mechanism, where each token attends to all other tokens.
- The figure (b) depicts Sliding Window Attention, where each token attends to a fixed range of tokens before and after it.
- The figure (c) illustrates Dilated Sliding Window Attention, where each token attends to tokens at fixed intervals before and after it.
- The figure (d) demonstrates Global+Sliding Window Attention, where, in addition to each token attending to a fixed range of tokens, a few “specific” tokens attend to all other tokens mutually.
- The marked positions near the beginning and middle are illustrative. The downstream task determines which tokens receive global attention.
Attention connections scale as for window width and global tokens. They are linear in length only while those settings stay bounded.
Autoregressive Modeling
The autoregressive experiments use sliding windows with dilation on some heads and a causal mask, so each position reads only the past. The bidirectional diagram cannot be used unchanged for next-token prediction.
Specifically, smaller dilation rates are used in the lower layers, while larger dilation rates are used in the higher layers, allowing the model to have different receptive fields at different levels.
Some training techniques are also employed. Instead of directly using ultra-long contexts from the beginning, the context length is gradually increased, making it easier for the model to converge.
Starting from shorter sequence lengths and window sizes in the initial stage helps the model quickly learn local details. In each subsequent stage, the authors double both the window size and sequence length, and halve the learning rate, leaving the costly long inputs to the end. This speeds up training and prepares the model to handle longer sequences.
The model is trained in five stages. In these five stages, the model gradually adapts to processing increasingly long sequences.
The five-stage schedule starts at 2,048 and ends at 23,040, constrained by GPU memory. It is not the final stage alone that starts at 2,048.
Autoencoding Modeling
In addition to comparing autoregressive models, the authors also compare with autoencoding models like BERT.
Due to the high cost of training BERT, the authors choose to continue MLM pretraining from RoBERTa while modifying the original attention mechanism to the Longformer design.
Here, the Sliding Window Attention mechanism is used, as shown in figure (b).
This training and downstream setup extends the context from RoBERTa's 512 to 4,096 tokens. The authors copy RoBERTa's learned position embeddings to initialize additional positions, then continue MLM pre-training; replacing attention alone would leave the position range too short.
Some tasks need global attention: classification assigns it to [CLS], QA to question tokens, and WikiHop also to answer candidates. A global token attends to every token and receives attention back; global attention uses separate Q/K/V projections. This is Figure 2(d). Adding arbitrarily many global tokens would change the cost.
Discussion
Autoregressive Experiment Results
For the small models in Table 2, Longformer reaches 1.10 test BPC on text8 and 1.00 on enwik8, the lowest in those comparison groups.
Table 3 tests only enwik8: Longformer has 102M parameters and 0.99 BPC, matching the roughly 100M-parameter Sparse Transformer at 0.99. The 209M-parameter Adaptive Span reaches 0.98, and the 277M-parameter Compressive Transformer reaches 0.97. Longformer is not the best large-model BPC in the table.
Autoencoding Experiment Results
The table above summarizes the results of all fine-tuning experiments.
Longformer-base beats RoBERTa-base on all six development sets. WikiHop rises 72.4→75.0 and Hyperpartisan 87.4→94.8, while OntoNotes 78.4→78.6 and IMDB 95.3→95.7 change less. Their metrics differ, so these gains should not be added or all called accuracy gains.
- For tasks requiring long contexts (e.g., WikiHop and Hyperpartisan), the performance improvement is particularly significant.
- For TriviaQA, the improvement is smaller because local context is usually sufficient to answer the questions.
- For HotpotQA, supporting fact supervision allows the model to easily find relevant context and then focus on the local context, resulting in smaller gains.
- WikiHop only includes distant supervision for intermediate reasoning chains, and Longformer excels by reasoning over the entire context.
- IMDB is sentiment classification, where mostly short documents limit long-context gains. OntoNotes is coreference resolution: nearby mentions can often be linked by the chunked baseline. The coreference explanation does not describe IMDB.
Autoregressive Ablation Experiments
The authors conducted two sets of ablation experiments on autoregressive models.
The first set tests the sliding window design:
- Decreasing w (from 512 to 32), meaning gradually reducing the sliding window size from the bottom layer.
- Fixed w (=230), meaning using the same sliding window size from start to finish.
- Increasing w (from 32 to 512), meaning gradually increasing the sliding window size from the bottom layer.
Within this partial run, increasing gives 1.21 BPC, below fixed at 1.23 and decreasing at 1.24.
With dilation on two attention heads, development BPC falls from 1.21 to 1.20. The gain is small and belongs to this 150K-step ablation.
Autoencoding Ablation Experiments
The authors conducted a series of ablation experiments on the autoencoding model.
First, all experiments were conducted using the Longformer-base model. Unless otherwise specified, all experiments use the same hyperparameters and fine-tune for five epochs.
- Longer input: reducing the limit to 2,048 tokens lowers WikiHop from 73.8 to 73.1 in this setting.
- Global attention and projections: removing separate global projections lowers 73.8→72.2; removing global attention as well yields 65.5. The latter row changes two factors, so all 8.3 points cannot be attributed to just one.
- MLM continuation: omitting it lowers 73.8→73.2 under this initialization and task setup.
- Fine-tuning time: extending five epochs to fifteen raises 73.8→75.0; this row changes the schedule.
- RoBERTa-base control: at length 512 with full attention, Longformer scores 71.7 versus RoBERTa-base 72.4. This supports the longer-context explanation without eliminating every training difference.
- Only new position embeddings trained: freezing RoBERTa during continued MLM and training just the new positions yields 73.5 after normal task fine-tuning, slightly below 73.8 with full continuation.
Conclusion
We believe the focus of this paper is on the design of sparse attention mechanisms.
Longformer lets the downstream task decide which tokens get a global view; the earlier Sparse Transformer uses fixed sparse connections to achieve something like global reach.
So which design is better? Longformer does better than several earlier sparse architectures in the reported tables, though some differences are small. These comparisons span different tasks and model sizes: it has the lowest BPC in the small text8/enwik8 groups, while on large enwik8 it matches Sparse Transformer and trails some larger models.
Therefore, the choice depends on the user’s needs (or trying out each architecture could also be considered).