Is normalization indispensable for training deep neural network?

By Jie Shao, Kaiqin Hu, Changhu Wang, Xiangyang Xue, Bhiksha Raj. 2020

68 citations recorded by the source corpus.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

1. Soham De
2. Andrew Brock
3. Samuel L. Smith
4. Karen Simonyan
5. Quoc V. Le
6. Mingxing Tan
7. Alexandre Sablayrolles
8. Herv'e J'egou

Citations in the corpus

• High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Most of these works suppress the scale of the activations on the residual branch at initialization, by introducing either small constants or learnable scalars (Hanin & Rolnick, 2018; Zhang et al., 2019a; De & Smith, 2020; Shao et al., 2020).”
• Data Augmentation techniques in time series domain: a survey and taxonomy (2022, methodology): “Otherwise the learning of the ANN would be inefficient[115, 116] Although VAEs are algorithms capable of generating synthetic data, as opposed to traditional algorithms that only modify the original data, new NN models such as GANs have mitigated their use in the field because by nature they are cap”