RoFormer: Enhanced Transformer with Rotary Position Embedding

By Jianlin Su, Yu Lu, Murtadha H. M. Ahmed, Shengfeng Pan, Yu Lu, Bo Wen, Yunfeng Liu, Wen Bo. 2021

5,865 citations recorded by the source corpus.

Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “We use RoPE embeddings (Su et al., 2021) rather than absolute or relative position embeddings, since RoPE embeddings have been shown to have better performance on long sequence lengths.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We remove the absolute positional embeddings, and instead, add rotary positional embeddings (RoPE), introduced by Su et al. (2021), at each layer of the network.”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We use the standard transformer architecture (Vaswani et al., 2017), apply pre-normalization using RMSNorm (Zhang and Sennrich, 2019), use the SwiGLU activation function (Shazeer, 2020), and rotary positional embeddings (RoPE, Su et al. 2022).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “For example, alternative positional embedding schemes (Su et al., 2021; Press et al., 2021) or novel activation functions (Shazeer, 2020).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “, 2021, Biderman, 2021, Wang and Komatsuzaki, 2021], we use rotary embeddings [Su et al., 2021] instead of the learned positional embeddings that OpenAI’s GPT models use [Radford et al.”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “For the positional encoding, we adopt rotary position embedding (Su et al., 2021).”
Code Llama: Open Foundation Models for Code (2023, methodology): “For long context fine-tuning (LCFT), we use a learning rate of 2 e − 5 , a sequence length of 16,384, and reset RoPE frequencies with a base value of θ = 10 6 .”
Llemma: An Open Language Model For Mathematics (2023, methodology): “Due to compute constraints, we were unable to verify that training L LEMMA 34B with a contracted RoPE base period did not come with a performance penalty, therefore for that model we preserved θ = 1 , 000 , 000 .”
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “In our experiments, we use Transformer models with Rotary positional encoding [32] which adds the position information to the key and query vectors just before computing the attention.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “We use rotary embeddings introduced by Su et al. (2021) and now in widespread use (Black et al., 2022; Chowdhery et al., 2022; Zeng et al., 2022) as our positional embedding type of choice.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “For PEs in GLM-130B, we adopt Rotary Positional Encoding (RoPE, Su et al. (2021)) rather than ALiBi (Press et al., 2021).”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[104] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “Firstly, GPT-J replaces learned positional embeddings with rotary positional embeddings [49], a form of relative positional embedding.”
Evolutionary-scale prediction of atomic level protein structure with a language model (2022, background): “[66] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu.”
SoundStorm: Efficient Parallel Audio Generation (2023, positive): “We use a Conformer with 350M parameters, with 12 layers, 16 attention heads, embedding size and model dimension of 1024, feedforward dimension of 4096, convolution kernel size of 5, and rotary positional embeddings (Su et al., 2021).”
What Language Model to Train if You Have One Million GPU Hours? (2022, background): “Su et al. (2021) proposed rotary embeddings, where the query and key representations inside the self-attention mechanism are modiﬁed such that the attention captures relative distances between them.”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “We encode the position of tokens using rotary position encoding (Su et al., 2021).”