Root Mean Square Layer Normalization

By Biao Zhang, Rico Sennrich. 2019

1,797 citations recorded by the source corpus.

Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these improvements expensive and significantly slows the underlying network, e.g. RNN in particular. In this paper, we hypothesize that re-centering invariance in LayerNorm is dispensable and propose root mean square layer normalization, or RMSNorm. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm. We also present partial RMSNorm, or pRMSNorm where the RMS is estimated from p% of the summed inputs without breaking the above properties. Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%~64% on different models. Source code is available at https://github.com/bzhangGo/rmsnorm.

Subfields: Artificial Intelligence, Neurology, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Neurology

In Model Architectures

In Optimization & Training

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We use the RMSNorm normalizing function, introduced by Zhang and Sennrich (2019).”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “We use RMSNorm as in (Zhang & Sennrich, 2019; Shazeer et al., 2018b) instead of standard LayerNorm (Ba et al., 2016).”
Improving language models by retrieving from trillions of tokens (2021, methodology): “, 2019), with some minimal changes: we replace LayerNorm with RMSNorm (Zhang and Sennrich, 2019) and use relative position encodings (Dai et al.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “• Transformer++: A Transformer with the following enhancements: RMS normalization [47], Swish activation [48] and a GLU multiplicative branch [38] in the feed forward inverted bottleneck (SwiGLU) [39].”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “We explored “RMS (root-mean-square) norm” (Zhang and Sennrich, 2019) as an alternative to layer normalization as well as the Rezero (Bachlechner et al., 2020) initialization scheme, including combining Rezero with Layer Norm and RMS Norm.”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, neutral): “…Flow [24] 17.20 + torch.compile & bf16 [38] - 100k × 1024 16.59 + AdamW smaller beta2 [1] 16.61 + Logit Normal Sampling [8] 13.99 + Velocity Direction Loss [43] 12.52 + Gradient Clip (1.0) [5] 14.40 + SwiGLU FFN [35] 10.10 + RMS Norm [46] 9.25 + Rotary Pos Embed [36] 7.13 + f16 & patchsize=1 5.58”
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, positive): “We employ RMSNorm (Zhang & Sennrich, 2019) to normalize the input of each transformer sub-layer, thereby enhancing training stability.”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, positive): “We fold the pre-MLP RMS norm gain parameters (Zhang and Sennrich (2019), Section 3) into the MLP input matrices, as described in (Gurnee et al. (2024), Appendix A.1) and then train the transcoder on input activations just after the pre-MLP RMSNorm, to reconstruct the MLP sublayer’s output as the…”
Uncertainty Estimation for Language Reward Models (2022, methodology): “In common with Gopher, our model architecture is similar to GPT-2 [32], with the following differences: • The models use root-mean-square layer normalization [51, RMSNorm] instead of layer normalization [2, LayerNorm].”
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (2024, background): “Most contemporary LLMs are composed of two core components: token-wise modules (including MLP and normalization (Ba et al., 2016; Zhang & Sennrich, 2019)) and attention (Vaswani et al., 2023) modules.”
Gemma: Open Models Based on Gemini Research and Technology (2024, methodology): “No-tably, the 7B model uses multi-head attention while the 2B checkpoints use multi-query attention (with 𝑛𝑢𝑚 _ 𝑘𝑣 _ ℎ𝑒𝑎𝑑𝑠 = 1), based on ablations that showed that multi-query attention works well at small scales (Shazeer, 2019).”
OLMo: Accelerating the Science of Language Models (2024, methodology): “We believe this was the safest option and it was also the fastest compared to the other variants we considered: parametric layer norm and RMSNorm (Zhang and Sennrich, 2019).”
ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “Referring to the Feed-Forward Network (FFN) in LLaMA [11], we adopt cascaded blocks of RMSNorm [43], SiLU activation functions [44], and residual connections [45].”
Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “Following PaLM (Chowdhery et al., 2022), the bias terms were removed from the QKV projections and all Layer-Norms were applied without bias and centering (Zhang & Sennrich, 2019).”
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2024, positive): “We follow the ex-periment setup from Lialin et al. (2023), which adopts a LLaMA-based 3 architecture with RMSNorm and SwiGLU activations (Touvron et al., 2023; Zhang & Sennrich, 2019; Shazeer, 2020).”
Emerging Properties in Unified Multimodal Pretraining (2025, positive): “It adopts RMSNorm [98] for normalization, SwiGLU [65] for activation, RoPE [67] for positional encoding, and GQA [2] for KV cache reduction.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “Additionally, Root Mean Square Layer Normalization (RM-SNorm) [211] is adopted in LLaMA as a replacement for Layer Normalization to ensure efficient training.”