Understanding and Improving Layer Normalization

By Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, Junyang Lin. 2019

543 citations recorded by the source corpus.

Layer normalization (LayerNorm) is a technique to normalize the distributions of intermediate layers. It enables smoother gradients, faster training, and better generalization accuracy. However, it is still unclear where the effectiveness stems from. In this paper, our main contribution is to take a step further in understanding LayerNorm. Many of previous studies believe that the success of LayerNorm comes from forward normalization. Unlike them, we find that the derivatives of the mean and variance are more important than forward normalization by re-centering and re-scaling backward gradients. Furthermore, we find that the parameters of LayerNorm, including the bias and gain, increase the risk of over-fitting and do not work in most cases. Experiments show that a simple version of LayerNorm (LayerNorm-simple) without the bias and gain outperforms LayerNorm on four datasets. It obtains the state-of-the-art performance on En-Vi machine translation. To address the over-fitting problem, we propose a new normalization method, Adaptive Normalization (AdaNorm), by replacing the bias and gain with a new transformation function. Experiments show that AdaNorm demonstrates better results than LayerNorm on seven out of eight datasets.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Lei He
2. Sheng Zhao
3. Yanqing Liu
4. Yu Wu
5. Shujie Liu
6. Jinyu Li
7. Xipeng Qiu
8. Furu Wei

In Optimization & Training

1. Michael W. Mahoney
2. A. Gholami
3. Kurt Keutzer
4. Zi-Mei Yao
5. Zhuo Chen
6. Sheng Shen
7. Xu Sun
8. Ling Shao

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “The current stage i is injected into the network with Adaptive Layer Normalization [Xu et al., 2019] operator, i.”
• Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, methodology): “For example, LR is suitable for recurrent neural networks and language applications (Ba et al., 2016; Xu et al., 2019), while IN is widely used in image stylization and image generation (Ulyanov et al., 2016; 2017).”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “Note that φ ( (cid:99) X ) is treated as a changing constant (not a function) and the gradient of φ ( (cid:99) X ) is detached in the implementation [27].”
• Rethinking Skip Connection with Layer Normalization (2020, methodology): “In recent years, a large number of skip connection methods have been used for neural networks (He et al., 2016a; Srivastava et al., 2015; He et al., 2016b; Vaswani et al., 2017; Szegedy et al., 2017; Wang et al., 2019; Nguyen and Salazar, 2019; Xu et al., 2019; Bachlechner et al., 2020; Liu et al., ”
• BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, background): “More recently, LayerNorm alternatives have been proposed and shown to have better gradient propagation through the network (Xu et al., 2019; Shen et al., 2020).”
• Dual PatchNorm (2023, background): “Xu et al. (2019) analyze LayerNorm and show that the derivatives of mean and variance have a greater contribution to final performance as opposed to forward normalization.”
• A Survey of Transformers (2021, background): “Substitutes AdaNorm[152], scaled l2 normalization[93], PowerNorm[120]”
• A Survey on Vision Transformer (2020, unknown): “Some other normalization algorithms [249], [198], [5] have been proposed to improve training of transformer.”
• Mitigating Neural Network Overconfidence with Logit Normalization (2022, methodology): “…the literature, normalization has been widely used in metric learning (Sohn, 2016; Wu et al., 2018; van den Oord et al., 2018), face recognition (Ranjan et al., 2017; Liu et al., 2017; Wang et al., 2017; 2018; Deng et al., 2019; Zhang et al., 2019), and self-supervised learning (Chen et al., 2020).”
• PowerNorm: Rethinking Batch Normalization in Transformers (2020, background): “The recent work of (Xu et al., 2019) studies why LN helps training, and in particular it finds that the derivatives of LN help recenter and rescale backward gradients.”
• Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, unknown): “We also disable all bias terms in Layer Norms (Xu et al., 2019).”