Improving Transformer Optimization Through Better Initialization

By Xiao Shi Huang, Felipe Pérez, Jimmy Ba, Maksims Volkovs. 2020

186 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Li Dong
2. Furu Wei
3. Shaohan Huang
4. Shuming Ma
5. Hongyu Wang
6. Mohit Bansal
7. Zhe Gan
8. Luke S. Zettlemoyer

Citations in the corpus

• Unit Scaling: Out-of-the-Box Low-Precision Training (2023, methodology): “For an example of extending scale analysis to training, Huang et al. (2020) consider the training dynamics of a Transformer under Adam, using this to derive an initialisation scheme that avoids vanishing updates.”
• Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “In another study, a new weight initialization scheme is applied to improve Transformer optimization [114].”
• NormFormer: Improved Transformer Pretraining with Extra Normalization (2021, methodology): “Similarly, some other approaches targeted initialization as well, in particular ReZero (Bachlechner et al., 2020), FixUp (Huang et al., 2020) and LookLinear (Balduzzi et al., 2017).”
• Revisiting Deep Learning Models for Tabular Data (2021, background): “Due to the ubiquitous success of attention-based architectures for different domains (Dosovitskiy et al., 2021; Vaswani et al., 2017), several authors propose to employ attention-like modules for tabular DL as well (Arik and Pfister, 2020; Huang et al., 2020a; Song et al., 2019).”
• DeepNet: Scaling Transformers to 1,000 Layers (2022, methodology): “In order to alleviate the above issue, there have been efforts on improving the optimization of deep Transformer by means of better initialization [16], [17], [18], or better architecture [15], [19], [20], [21].”
• A Practical Survey on Faster and Lighter Transformers (2021, methodology): “[47] addressed two other issues: (i) Transformers are typically trained with optimizers that rely on adaptive learning rates as conventional SGD fails to train them efectively.”
• Escaping the Big Data Paradigm with Compact Transformers (2021, methodology): “[40] proposed Data-Efficient Image Transform-3 ers (DeiT).”
• Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster (2023, methodology): “…(Glorot & Bengio, 2010; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Yang & Schoenholz, 2018; Zhang et al., 2019; Bachlechner et al., 2020; Huang et al., 2020; Liu et al., 2020; Li et al., 2022). µP is the first comprehensive method to analytically control width-related training…”
• BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, methodology): “The transformer follows the standard implementation of Vaswani et. al [40], but with internal batch normalization layers removed and with an accompanying weight initialization scheme known as T-Fixup [60].”