DeepNet: Scaling Transformers to 1,000 Layers

By Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Furu Wei. 2022

261 citations recorded by the source corpus.

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Furu Wei
2. Li Dong
3. Luc Van Gool
4. Dacheng Tao
5. Shaohan Huang
6. Wenhui Wang
7. Trevor Darrell
8. Shuming Ma

In Electrical and Electronic Engineering

1. Yonghong Tian
2. Liuliang Yuan
3. Man Yao
4. Furu Wei
5. Boxing Xu
6. Zhaokun Zhou
7. Jiakui Hu
8. Guoqi Li

In Model Architectures

1. Tri Dao
2. Christopher R'e
3. Daniel Y. Fu
4. Stefano Ermon
5. Furu Wei
6. Atri Rudra
7. Wenhui Wang
8. Li Dong

In Scaling & Learning Theory

1. Li Dong
2. Atri Rudra
3. Etai Littwin
4. Christopher R'e
5. Tri Dao
6. Stefano Ermon
7. Yaru Hao
8. Yonghong Tian

Citations in the corpus

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “Transformers have grown larger [5] and deeper [87], but equipping them with longer context remains difficult [83], since the self-attention module at their heart has time and memory complexity quadratic in sequence length.”
• GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Fortunately, one of the attempts on Post-LN initialized with the newly-proposed DeepNorm (Wang et al., 2022b) generates promising training stability.”
• Language Models are General-Purpose Interfaces (2022, positive): “We use DeepNorm (Wang et al., 2022a) for Transformers.”
• Spike-driven Transformer (2023, background): “MS-Res-SNN has been proven [26] to satisfy dynamical isometry theory [72], which is a theoretical explanation of well-behaved deep neural networks [73, 74].”
• A Survey of Large Language Models (2023, unknown): “Compared with LN, DeepNorm [157] has shown a better capability to ensure the stability in training, which has been adopted by GLM-130B with post normalization.”
• H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, unknown): “…(Variš & Bojar, 2021; Zhang & Vaidya, 2021), a better initialization (Liu et al., 2020b;a; Xu et al., 2020; Zhu et al., 2021), customized optimizers (Cohen et al., 2022), improved normalization (Wang et al., 2022; Yang et al., 2022), weight decay (Loshchilov & Hutter, 2017), and early stopping.”
• A Comprehensive Overview of Large Language Models (2023, unknown): “DeepNorm [60] mitigates these adverse effects on the gradients.”
• BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “We also employ the DeepNorm [Wang et al., 2022a] for more stable pre-training.”
• Foundation Transformers (2022, methodology): “Following Wang et al. (2022a), the magnitude of attention output only depends on value and output projection: MSA ( X ) Therefore, for vanilla Pre-LN, the forward computation of the l -th sub-layer can be formulated as: We introduce two constants v l , w l to represent the scales of W l, 2 , W l, 1…”
• TorchScale: Transformers at Scale (2022, methodology): “To get rid of the pain when scaling up the models, T ORCH S CALE follows the theoretical derivation of DeepNet [Wang et al., 2022a] to improve the training stability.”
• Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, methodology): “For pre-LN models, we reproduced stable results and convergence, however the BLEU score we get is better (Table 11) than reported by Wang et al. (2022).”