The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformersroformer2021,
title = {{RoFormer: Enhanced Transformer with Rotary Position Embedding}},
author = {Jianlin Su and Yu Lu and Shengfeng Pan and Ahmed Murtadha and Bo Wen and Yunfeng Liu},
year = {2021},
eprint = {2104.09864},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2104.09864}
}
Compared with the name RoFormer, RoPE is probably more familiar.
RoPE, short for Rotary Position Embedding, is the core idea behind RoFormer and offers a new way to encode position.
Problem Definition
Pure self-attention cannot distinguish sequence order without positional signals or an ordering mechanism such as a causal mask. Positional encoding supplies that information; one common approach adds position vectors to token embeddings.
BERT and early GPT models learn a positional table of shape N×dmodel. This is straightforward, but positions beyond N need an extension or adaptation method. Changing input length within the existing range does not require retraining.
Transformer-XL does not add positional encoding to the value term. Later methods such as T5 make a similar choice, but this is not a universal rule for all subsequent research.
Previous research shows that absolute and relative positional encodings have their own advantages and disadvantages. This study aims to propose a method that integrates both absolute and relative positional encoding.
Model Architecture
Language modeling based on Transformers typically uses self-attention mechanisms to utilize the positional information of each token. The authors aim for the dot product to encode positional information only in a relative form:
⟨fq(xm,m),fk(xn,n)⟩=g(xm,xn,m−n)
The ultimate goal is to find an equivalent encoding mechanism to solve the functions fq(xm,m) and fk(xn,n) to satisfy the above relation. After a series of derivations (details in the paper), the proposed solution is:
fq(xm,m)=(Wqxm)eimθ
fk(xn,n)=(Wkxn)einθ
g(xm,xn,m−n)=ℜ[(Wqxm)(Wkxn)∗ei(m−n)θ]
This is the two-dimensional complex representation: each projected vector corresponds to a complex number. ℜ takes the real part, ∗ the conjugate, and θ is a preset nonzero rotation frequency.
Higher-Dimensional Extension
The diagram above illustrates RoPE.
To generalize the results from a 2D space to any xi∈Rd where d is even, the authors divide the d-dimensional space into d/2 subspaces, converting f{q,k} into:
Here RΘ,n−md=(RΘ,md)⊤RΘ,nd. Orthogonal rotation preserves vector norms. We also restore the m subscript and transpose on Wq omitted on the right of the paper’s Eq. (16), making it consistent with the left-hand expansion.
Long-Range Decay
The authors use geometrically spaced frequencies, written here as θi=10000−2i/d with zero-based indexing, equivalent to the one-based formula above. Their Abel-transform analysis bounds the inner product. The plotted relative upper bound falls overall, with visible fluctuations, matching the intuition that distant words often have weaker relations. This does not make every query–key score decrease monotonically with distance or guarantee unlimited length extrapolation.
Discussion
Comparison with Other Positional Encodings
The above image compares RoPE with other positional encoding methods. On the left, compared with BERT, the model using RoPE shows better performance during training, with a faster decrease in MLM Loss. On the right, adding RoPE to Performer shows faster convergence and better performance at the end of training.
GLUE Benchmark
The authors fine-tuned with Hugging Face Transformers. In Table 2, RoFormer scores higher on MRPC, STS-B and QQP, but lower on SST-2, QNLI and MNLI. The six GLUE datasets therefore show a trade-off, not a majority or universal win over BERT.
Limitations
Despite strong theoretical support and encouraging experimental results, the authors acknowledge some limitations:
The mathematical representation of relative positional relationships as rotations in 2D subspaces lacks a deep explanation for why this method results in faster convergence than baseline models with other positional encoding strategies.
The distance-decay analysis helps explain positional structure, but does not fully explain the long-text results or guarantee performance beyond training lengths.
Conclusion
This paper may be challenging to read, but RoPE is fascinating. We highly recommend reading the original paper. Many later works, including LLaMA, have adopted RoPE. Subsequent work has explored and improved upon it, and we will discuss those developments as we encounter them.