PowerNorm: Rethinking Batch Normalization in Transformers

By Sheng Shen, Zi-Mei Yao, A. Gholami, Michael W. Mahoney, Kurt Keutzer. 2020

90 citations recorded by the source corpus.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Dacheng Tao
2. Yunhe Wang
3. Kai Han
4. Soham De
5. Andrew Brock
6. Samuel L. Smith
7. Jianyuan Guo
8. Chunjing Xu

In Optimization & Training

1. Soham De
2. Andrew Brock
3. Samuel L. Smith
4. Karen Simonyan
5. Yunhe Wang
6. Kai Han
7. Chunjing Xu
8. Yehui Tang

Citations in the corpus

• High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “The performance of batch-normalized networks can also degrade if the batch statistics have a large variance during training (Shen et al., 2020).”
• A Survey of Transformers (2021, background): “Substitutes AdaNorm[152], scaled l2 normalization[93], PowerNorm[120]”
• BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, background): “More recently, LayerNorm alternatives have been proposed and shown to have better gradient propagation through the network (Xu et al., 2019; Shen et al., 2020).”
• A Transformer-based Framework for Multivariate Time Series Representation Learning (2020, background): “Additionally, the inferior performance of batch normalization in NLP has been mainly attributed to extreme variation in sample length (i.e., sentences in most tasks) (Shen et al., 2020), while in the datasets we examine this variation is much smaller.”
• Rethinking Skip Connection with Layer Normalization (2020, methodology): “In recent years, a large number of skip connection methods have been used for neural networks (He et al., 2016a; Srivastava et al., 2015; He et al., 2016b; Vaswani et al., 2017; Szegedy et al., 2017; Wang et al., 2019; Nguyen and Salazar, 2019; Xu et al., 2019; Bachlechner et al., 2020; Liu et al., ”
• A Survey on Vision Transformer (2020, background): “Some other normalization algorithms [249], [198], [5] have been proposed to improve training of transformer.”