Transformers without Tears: Improving the Normalization of Self-Attention

By Toan Q. Nguyen, Julian Salazar. 2019

278 citations recorded by the source corpus.

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PRENORM) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose l2 normalization with a single scale parameter (SCALENORM) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FIXNORM). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT '15 English-Vietnamese. We ob- serve sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT '14 English-German), SCALENORM and FIXNORM remain competitive but PRENORM degrades performance.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, critical): “Similar to [45], we also observed that using higher variance for weight initialization fails to converge.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “To stabilize the training of deeper Transformers, we train with LayerDrop (Fan et al., 2019) 0.05 and pre-normalization (Nguyen and Salazar, 2019).”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “We use prenorm residual units [21, 22] with dropout which helps training and regularizing deeper models.”
ReZero is All You Need: Fast Convergence at Large Depth (2020, methodology): “ iterations it takes for a 12 layer Transformer to reach 1.2 bits per byte (BPB) on enwiki8. Since the introduction of Transformers [27], there have been several competing placements of the LayerNorm [34, 35] within the Transformer to achieve better convergence [4, 36]. We experiment with 3 Transfor”
Masked Language Model Scoring (2019, methodology): “Our models are also described as baselines in a dedicated work (Nguyen and Salazar, 2019).”
Improving Transformer Models by Reordering their Sublayers (2019, background): “and repositioning the layer normalization (Nguyen and Salazar, 2019) can also have a positive effect on performance.”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “Training FlauBERTLARGE Training very deep Transformers is known to be susceptible to instability (Wang et al., 2019b; Nguyen and Salazar, 2019; Xu et al., 2019; Fan et al., 2019).”
Rethinking Skip Connection with Layer Normalization (2020, methodology): “The evaluation metric is BLEU (Papineni et al., 2002) against the reference translations.”
Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, result): “At the same time, pre-LN is stable as reported by, e.g., Nguyen & Salazar (2019); Wang et al. (2022); Liu et al. (2020a): we are able to reduce warmup from 64k to 16k, increase learning rate from 0.03 to 0.5, and obtain better results than the training setting from the post-LN baseline.”
A Comprehensive Overview of Large Language Models (2023, unknown): “MT-NLG [21] found higher variance for weight initialization leads to unstable training, hence validating small initialization scheme [369].”
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, background): “Nguyen and Salazar (2019) inject multiple normalization blocks in speciﬁc network submodules to improve model performance.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, unknown): “tigated in self-attention of transformers [103].”
Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, methodology): “Following the original Transformer structure, the input is prenormalized passing through a Batch Normalization layer [41], [42], and skip connections are used to sum the input to the output of the SSA module (see Figure 3).”
Improving Transformer Optimization Through Better Initialization (2020, methodology): “…architectures are large and require many GPU hours to train (Yang et al., 2019; Lan et al., 2020), recent research has focused on understanding and improving optimization in these models (Ahmed et al., 2017; Chen et al., 2018b; Nguyen & Salazar, 2019; Aji & Heaﬁeld, 2019; Liu et al., 2020).”
Revisiting Deep Learning Models for Tabular Data (2021, methodology): “The PreNorm variant is known for better optimization properties as opposed to the original Transformer, which is a PostNorm-Transformer (Liu et al., 2020; Nguyen and Salazar, 2019; Wang et al., 2019b).”