Back to paper library
arXiv 2021Neurocomputing 2024Language & TransformersPaper notes

RoFormer

Rotary Position Embedding

Original paper

RoFormer: Enhanced Transformer with Rotary Position Embedding

Paper authors

Jianlin Su · Yu Lu · Shengfeng Pan · Ahmed Murtadha · Bo Wen · Yunfeng Liu

Publication
Neurocomputing 2024Journal
First submitted to arXiv
arXiv:2104.09864
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformersroformer2021,
  title = {{RoFormer: Enhanced Transformer with Rotary Position Embedding}},
  author = {Jianlin Su and Yu Lu and Shengfeng Pan and Ahmed Murtadha and Bo Wen and Yunfeng Liu},
  year = {2021},
  eprint = {2104.09864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.09864}
}
On this page

Compared with the name RoFormer, RoPE is probably more familiar.

RoPE, short for Rotary Position Embedding, is the core idea behind RoFormer and offers a new way to encode position.

Problem Definition

Pure self-attention cannot distinguish sequence order without positional signals or an ordering mechanism such as a causal mask. Positional encoding supplies that information; one common approach adds position vectors to token embeddings.

Absolute Position: Sinusoidal

In the first Transformer paper, a sinusoidal function was used:

PE(pos,k)={sin⁡(pos/100002i/dmodel)if k=2icos⁡(pos/100002i/dmodel)if k=2i+1PE_{(pos, k)} = \begin{cases} \sin(pos/10000^{2i/d_{\text{model}}}) & \text{if } k = 2i \\ \cos(pos/10000^{2i/d_{\text{model}}}) & \text{if } k = 2i + 1 \end{cases}

The function depends on the position index and feature dimension, not the total sequence length. It can be evaluated beyond training lengths.

Absolute Position: Learned

BERT and early GPT models learn a positional table of shape N×dmodelN \times d_{\text{model}}. This is straightforward, but positions beyond NN need an extension or adaptation method. Changing input length within the existing range does not require retraining.

Relative Position: XL Style

In this paper, the self-attention mechanism's QKTQK^T is expanded:

  • Qi=(xi+posi)WQQ_i=(x_i+pos_i)W_Q
  • Kj=(xj+posj)WKK_j=(x_j+pos_j)W_K
  • QiKjT=(xi+posi)WQWKT(xj+posj)TQ_iK_j^T=(x_i+pos_i)W_QW_K^T(x_j+pos_j)^T

This results in:

  • QiKjT=xiWQWKTxjT+xiWQWKTposjT+posiWQWKTxjT+posiWQWKTposjTQ_iK_j^T=x_iW_QW_K^Tx_j^T+x_iW_QW_K^Tpos_j^T+pos_iW_QW_K^Tx_j^T+pos_iW_QW_K^Tpos_j^T

Then, posjpos_j is replaced with relative positional vector Ri−jR_{i-j}, and posipos_i is replaced with learnable vectors uu and vv:

  • QiKjT=xiWQWKTxjT+xiWQWKTRi−jT+uWQWKTxjT+vWQWKTRi−jTQ_iK_j^T=x_iW_QW_K^Tx_j^T+x_iW_QW_K^T{\color{red}{R_{i-j}^T}}+{\color{green}{u}}W_QW_K^Tx_j^T+{\color{green}{v}}W_QW_K^T{\color{red}{R_{i-j}^T}}

Absorb uWQuW_Q and vWQvW_Q separately into new learned vectors named uu and vv. They remain distinct parameters:

  • QiKjT=xiWQWKTxjT+xiWQWKTRi−jT+uWKTxjT+vWKTRi−jTQ_iK_j^T=x_iW_QW_K^Tx_j^T+x_iW_QW_K^T{\color{red}{R_{i-j}^T}}+{\color{green}{u}}W_K^Tx_j^T+{\color{green}{v}}W_K^T{\color{red}{R_{i-j}^T}}

Considering the encoding space of Ri−jT{\color{red}{R_{i-j}^T}} differs from the original posjpos_j, WKTW_K^T is replaced with WK,RTW_{K, R}^T:

  • QiKjT=xiWQWKTxjT+xiWQWK,RTRi−jT+uWKTxjT+vWK,RTRi−jTQ_iK_j^T=x_iW_QW_K^Tx_j^T+x_iW_QW_{K, R}^T{\color{red}{R_{i-j}^T}}+{\color{green}{u}}W_K^Tx_j^T+{\color{green}{v}}W_{K, R}^T{\color{red}{R_{i-j}^T}}

Transformer-XL does not add positional encoding to the value term. Later methods such as T5 make a similar choice, but this is not a universal rule for all subsequent research.

Relative Position: T5 Style

In T5, the authors decoupled content and positional information, placing all position-related information into βi,j\beta_{i,j}:

  • QiKjT=xiWQWKTxjT+βi,jQ_iK_j^T=x_iW_QW_K^Tx_j^T + \beta_{i,j}

More Methods

We are not reviewing all positional encoding methods. For more information, refer to Su Jianlin's article:

Solution

Previous research shows that absolute and relative positional encodings have their own advantages and disadvantages. This study aims to propose a method that integrates both absolute and relative positional encoding.

Model Architecture

Language modeling based on Transformers typically uses self-attention mechanisms to utilize the positional information of each token. The authors aim for the dot product to encode positional information only in a relative form:

⟨fq(xm,m),fk(xn,n)⟩=g(xm,xn,m−n)\langle f_q(x_m, m), f_k(x_n, n) \rangle= g(x_m, x_n, m-n)

The ultimate goal is to find an equivalent encoding mechanism to solve the functions fq(xm,m)f_q(x_m, m) and fk(xn,n)f_k(x_n, n) to satisfy the above relation. After a series of derivations (details in the paper), the proposed solution is:

  • fq(xm,m)=(Wqxm)eimθf_q(x_m, m) = (W_qx_m)e^{im\theta}
  • fk(xn,n)=(Wkxn)einθf_k(x_n, n) = (W_kx_n)e^{in\theta}
  • g(xm,xn,m−n)=ℜ[(Wqxm)(Wkxn)∗ei(m−n)θ]g(x_m, x_n, m - n) = \Re \left[ (W_q x_m)(W_k x_n)^* e^{i(m - n) \theta} \right]

This is the two-dimensional complex representation: each projected vector corresponds to a complex number. ℜ\Re takes the real part, ∗* the conjugate, and θ\theta is a preset nonzero rotation frequency.

Higher-Dimensional Extension

The diagram above illustrates RoPE.

To generalize the results from a 2D space to any xi∈Rdx_i \in R^d where dd is even, the authors divide the dd-dimensional space into d/2d/2 subspaces, converting f{q,k}f_{\{q,k\}} into:

f{q,k}(xm,m)=RΘ,mdW{q,k}xmf_{\{q,k\}}(x_m, m) = R^d_{\Theta,m} W_{\{q,k\}} x_m

where

RΘ,md=(cos⁡mθ1−sin⁡mθ100⋯00sin⁡mθ1cos⁡mθ100⋯0000cos⁡mθ2−sin⁡mθ2⋯0000sin⁡mθ2cos⁡mθ2⋯00⋮⋮⋮⋮⋱⋮⋮0000⋯cos⁡mθd/2−sin⁡mθd/20000⋯sin⁡mθd/2cos⁡mθd/2)R^d_{\Theta,m} = \begin{pmatrix} \cos m\theta_1 & -\sin m\theta_1 & 0 & 0 & \cdots & 0 & 0 \\ \sin m\theta_1 & \cos m\theta_1 & 0 & 0 & \cdots & 0 & 0 \\ 0 & 0 & \cos m\theta_2 & -\sin m\theta_2 & \cdots & 0 & 0 \\ 0 & 0 & \sin m\theta_2 & \cos m\theta_2 & \cdots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & 0 & 0 & \cdots & \cos m\theta_{d/2} & -\sin m\theta_{d/2} \\ 0 & 0 & 0 & 0 & \cdots & \sin m\theta_{d/2} & \cos m\theta_{d/2} \end{pmatrix}

is the rotation matrix with predetermined parameters Θ={θi=10000−2(i−1)/d,i∈[1,2,...,d/2]}\Theta = \{\theta_i = 10000^{-2(i-1)/d}, i \in [1, 2, ..., d/2]\}.

Applying RoPE to the self-attention mechanism's equation, we get:

qm⊤kn=(RΘ,mdWqxm)⊤(RΘ,ndWkxn)=xm⊤Wq⊤RΘ,n−mdWkxnq_m^\top k_n = (R^d_{\Theta,m} W_q x_m)^\top (R^d_{\Theta,n} W_k x_n) = x_m^\top W_q^\top R^d_{\Theta,n-m} W_k x_n

Here RΘ,n−md=(RΘ,md)⊤RΘ,ndR^d_{\Theta,n-m} = (R^d_{\Theta,m})^\top R^d_{\Theta,n}. Orthogonal rotation preserves vector norms. We also restore the mm subscript and transpose on WqW_q omitted on the right of the paper’s Eq. (16), making it consistent with the left-hand expansion.

Long-Range Decay

The authors use geometrically spaced frequencies, written here as θi=10000−2i/d\theta_i=10000^{-2i/d} with zero-based indexing, equivalent to the one-based formula above. Their Abel-transform analysis bounds the inner product. The plotted relative upper bound falls overall, with visible fluctuations, matching the intuition that distant words often have weaker relations. This does not make every query–key score decrease monotonically with distance or guarantee unlimited length extrapolation.

Discussion

Comparison with Other Positional Encodings

The above image compares RoPE with other positional encoding methods. On the left, compared with BERT, the model using RoPE shows better performance during training, with a faster decrease in MLM Loss. On the right, adding RoPE to Performer shows faster convergence and better performance at the end of training.

GLUE Benchmark

The authors fine-tuned with Hugging Face Transformers. In Table 2, RoFormer scores higher on MRPC, STS-B and QQP, but lower on SST-2, QNLI and MNLI. The six GLUE datasets therefore show a trade-off, not a majority or universal win over BERT.

Limitations

Despite strong theoretical support and encouraging experimental results, the authors acknowledge some limitations:

  1. The mathematical representation of relative positional relationships as rotations in 2D subspaces lacks a deep explanation for why this method results in faster convergence than baseline models with other positional encoding strategies.
  2. The distance-decay analysis helps explain positional structure, but does not fully explain the long-text results or guarantee performance beyond training lengths.

Conclusion

This paper may be challenging to read, but RoPE is fascinating. We highly recommend reading the original paper. Many later works, including LLaMA, have adopted RoPE. Subsequent work has explored and improved upon it, and we will discuss those developments as we encounter them.