Back to paper library
arXiv 2019ACL 2019Language & TransformersPaper notes

Transformer-XL

Longer Contexts

Original paper

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Paper authors

Zihang Dai · Zhilin Yang · Yiming Yang · Jaime Carbonell · Quoc V. Le · Ruslan Salakhutdinov

Publication
ACL 2019Conference
First submitted to arXiv
arXiv:1901.02860
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformerstransformerxl2019,
  title = {{Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context}},
  author = {Zihang Dai and Zhilin Yang and Yiming Yang and Jaime Carbonell and Quoc V. Le and Ruslan Salakhutdinov},
  year = {2019},
  eprint = {1901.02860},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.02860}
}
On this page

BERT has a maximum supported sequence length, not a requirement that every input have exactly the same length. Transformer-XL studies autoregressive language modeling; BERT is an example of a context limit, not the model being directly extended here. But if we could ignore that limit, how would we ideally feed a text to the model?

  • Input the entire text at once, however long it is. Perfect!

Defining the Problem

But that's not feasible.

Reality is harsh, with limited computing power and memory. We can't input all the text at once. The authors identify several specific issues:

Computational Resource Constraints

Training large models requires substantial computational resources, including high-performance GPUs or TPUs. Memory limits the text length that can be processed at once; too long texts cause memory overflow or excessive computation time. The attention mechanism of Transformers uses memory that grows quadratically with the context length, making it difficult to handle long texts.

Context Fragmentation Problem

Figure 1 shows the vanilla Transformer: training does not pass state across segments, while the evaluation window shifts one token and is recomputed for each prediction.

To fit within computational limits, texts are usually segmented into fixed-length chunks. This approach results in the loss of contextual information, hindering the model's ability to make accurate predictions. Fixed-length contexts restrict the model's predictive capacity, leading to suboptimal performance. The lack of cross-segment context information decreases the model's optimization efficiency, failing to capture long-term dependencies in the text.

Solving the Problem

Model Architecture

The dashed box holds old states with gradients stopped. Queries from the new segment can attend to keys and values from both the old and current segments.

The authors first address the context fragmentation problem by introducing a recurrence mechanism similar to those used in RNN models. In training, the hidden states from the previous segment are stored and reused when processing the next segment. This allows the model to leverage information from previous segments, effectively capturing long-term dependencies and avoiding context fragmentation.

When processing A then B, B can attend to cached hidden states from A and, if memory permits, earlier segments. These are past segments, not future text. Cached states use stop-gradient: forward information crosses segment boundaries, while gradients do not backpropagate through the entire document.

State reuse also speeds evaluation. At the longest tested attention length of 3,800, the paper reports a 1,874× per-token speedup on enwik8 against a sliding-window baseline that recomputes the full window for each prediction. This is neither a universal speedup over every Transformer implementation nor a training-speed result.

New Challenges

In a standard Transformer model, sequence order information is provided by position encodings, which indicate each token’s position in the sequence and are input along with the token embeddings.

If we use the same position encodings in the recurrence mechanism, the model cannot differentiate between identical positions in different segments, leading to performance degradation.

For example, consider the sentence: "The cat sat on the mat." split into two segments:

  • Segment A: "The cat sat"
  • Segment B: "on the mat."

In a standard Transformer, each word's position is determined by its absolute position in the sentence. If we use the same position encodings for both segments, the model cannot distinguish the relative positions. "on" and "The" would both be encoded as position 1, "the" and "cat" as position 2, etc., confusing the model about their actual positions in the sequence.

Across the complete sentence, the positions would instead be "The" at 1, "cat" at 2, "sat" at 3, "on" at 4, "the" at 5, and "mat" at 6. Relative positions describe the distances between words: "cat" is one position after "The"; "sat" is two after "The" and one after "cat"; "on" is one after "sat" and three after "The". Those distances still make sense across the segment boundary.

Relative Position Encoding

The authors start by expanding the QKTQK^T attention score. In the article's row-vector notation, Qi=(xi+posi)WQQ_i=(x_i+pos_i)W_Q and Kj=(xj+posj)WKK_j=(x_j+pos_j)W_K, so

QiKjT=xiWQWKTxjT+xiWQWKTposjT+posiWQWKTxjT+posiWQWKTposjT.Q_iK_j^T=x_iW_QW_K^Tx_j^T+x_iW_QW_K^Tpos_j^T+pos_iW_QW_K^Tx_j^T+pos_iW_QW_K^Tpos_j^T.

The paper writes these four terms in column-vector notation:

These terms represent:

  • ExiTWqTWkExjE_{x_i}^T W_q^T W_k E_{x_j}: Content-based attention.
  • ExiTWqTWkUjE_{x_i}^T W_q^T W_k U_j: Attention related to the absolute position encoding of the key vector.
  • UiTWqTWkExjU_i^T W_q^T W_k E_{x_j}: Attention related to the absolute position encoding of the query vector.
  • UiTWqTWkUjU_i^T W_q^T W_k U_j: Attention related to the absolute position encoding of both query and key vectors.

This method has issues when reusing hidden states, as position encodings from different segments mix.

To resolve this, the authors replace the key's absolute posjpos_j with the relative vector Ri−jR_{i-j}, and replace the query's posipos_i with two learned vectors uu and vv:

QiKjT=xiWQWKTxjT+xiWQWKTRi−jT+uWQWKTxjT+vWQWKTRi−jT.Q_iK_j^T=x_iW_QW_K^Tx_j^T+x_iW_QW_K^TR_{i-j}^T+uW_QW_K^Tx_j^T+vW_QW_K^TR_{i-j}^T.

The products uWQuW_Q and vWQvW_Q can be absorbed separately into two learned vectors, still called uu and vv. The position vector has its own key projection, WK,RW_{K,R}, rather than sharing the content-key projection WKW_K. The resulting four terms are shown in the paper's notation:

Each term now has an intuitive meaning:

  • ExiTWqTWk,EExjE_{x_i}^T W_q^T W_{k,E} E_{x_j}: Content-based attention.
  • ExiTWqTWk,RRi−jE_{x_i}^T W_q^T W_{k,R} R_{i-j}: Relative position bias related to content.
  • uTWk,EExju^T W_{k,E} E_{x_j}: Trainable parameter uu controlling global content bias.
  • vTWk,RRi−jv^T W_{k,R} R_{i-j}: Trainable parameter vv controlling global position bias.

Improvements include:

  • Replacing Absolute Position Encoding: Replacing the absolute position encoding UjU_j used in terms (b) and (d) with their relative counterparts Ri−jR_{i-j}. This indicates that only relative distance matters for determining attention priors. RR is a sinusoidal encoding matrix without learnable parameters.
  • Introducing Trainable Parameters: Introducing trainable parameters uu and vv to replace the absolute position encoding of the query vector UiU_i in terms (c) and (d). This suggests that attention bias for different tokens should remain the same regardless of the query position.

Content and relative-position keys use separate projections, Wk,EW_{k,E} and Wk,RW_{k,R}. The vectors uu and vv are independent learned biases. Values do not receive this added relative-position term.

With these improvements, the authors developed the Transformer-XL architecture.

Discussion

Model Performance

The imported image combines Tables 1–4. WikiText-103 and One Billion Word use perplexity (PPL); enwik8 and text8 use bits per character (BPC). These are different metrics.

Transformer-XL was applied to word-level and character-level language modeling datasets, including WikiText-103, enwik8, text8, the One Billion Word Benchmark, and Penn Treebank.

  • Table 1: WikiText-103:

    • A major contemporary word-level benchmark with long-document context.
    • Contains 103M training tokens from 28K articles.
    • Attention length is 384 during training and 1600 during evaluation.
    • Uses adaptive softmax and input representations.
    • The previous best listed result is 20.5 PPL with 247M parameters; Transformer-XL Large reaches 18.3 with 257M. The Standard model has 24.0 with 151M, so model sizes should not be treated as one controlled ablation.
  • Table 2: enwik8:

    • Contains 100M bytes of unprocessed Wikipedia text.
    • Transformer-XL showed significant advantages over RNN-based models.
    • The 12-layer Transformer-XL gets 1.06 BPC with 41M parameters, matching the prior 64-layer Transformer at 1.06 with 235M—about 17% as many parameters.
    • The 18-layer model gets 1.03 with 88M; the 24-layer model gets 0.99 with 277M. Capacity changes too, so the full gap cannot be attributed to recurrence alone.
  • Table 3: text8:

    • Contains 100M processed Wikipedia characters with only 26 letters and spaces.
    • The best enwik8 model and hyperparameters are reused without further tuning; 24-layer Transformer-XL gets 1.08 BPC, versus 1.13 for the previous 64-layer Transformer in the table.
  • Table 4: One Billion Word Benchmark:

    • Tests short-term dependency modeling ability since sentences are shuffled.
    • Transformer-XL Large gets 21.8 PPL with 0.8B parameters, versus 23.7 with 1.0B for Adaptive Input; their model sizes and training setups are not identical.

Ablation Studies

The authors examined the effectiveness of the recurrence mechanism and the new position encoding scheme:

  1. Effectiveness of Relative Position Encoding:

    PPL init uses the training attention length; PPL best adjusts evaluation length. The first two blocks use 128M parameters, while the last uses 151M.

    Shaw et al. is another relative position scheme; Vaswani et al. and Al-Rfou et al. use absolute positions. Full scores every position in a segment and Half scores its later half. The paper says absolute schemes train better with half losses, but this table has no full-loss row for the same absolute scheme; the size of that effect cannot be read directly from it.

    Within the 128M rows using this paper's encoding and full losses, removing recurrence changes best PPL from 26.77 to 29.02. With recurrence kept, replacing the encoding with Shaw et al.'s changes 26.77 to 27.94, although evaluation attention lengths also differ. Keeping the encoding and recurrence but switching from full to half losses changes 26.77 to 28.33. The best row is recurrence plus this encoding plus full losses.

    For the 151M model, backpropagation still spans 128 positions. Evaluation attention lengths of 300 / 450 / 640 yield best PPL values 23.35 / 23.16 / 23.09. That is evidence about a longer evaluation context, not gradients flowing through 640 training positions.

  1. Effectiveness of the Recurrence Mechanism:

    The authors use One Billion Word, whose sentence order is shuffled, to focus on shorter dependencies. They train a 20-layer, roughly 0.3B-parameter model for 400K steps. Table 7 gives PPL 25.2 for their setup, 25.7 with Shaw et al.'s encoding and 27.1 without recurrence. Reusing state helps even here, consistent with reducing lost context at segment starts.

Conclusion

Transformer-XL shows strong language-modeling results. It handles longer dependencies more effectively than the RNN and vanilla Transformer baselines studied in the paper, while keeping past segments available as context.

It also evaluates much faster against the paper's recomputed-window baseline and can generate reasonably coherent long passages. To us, these strengths suggest room for applications in text generation, unsupervised feature learning, and perhaps image and speech modeling. The paper leaves plenty to explore in both improving existing tools and building new ones.