BLOCK-NORMALIZED GRADIENT METHOD: AN EMPIRICAL STUDY FOR TRAINING DEEP NEURAL NETWORK

By Adams Wei Yu, Lei Huang, Qihang Lin, Ruslan Salakhutdinov, Jaime G. Carbonell. 2018

36 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Bo Li
2. Quoc V. Le
3. Lei Huang
4. Ling Shao
5. David R. So
6. Chen Liang
7. Li Liu
8. Esteban Real

Citations in the corpus

• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “The normalized gradient is a commonly used heuristic in non-convex optimization [29, 41], which could help with the vanishing and exploding gradient problem [98, 63].”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “[53] were the first to propose block-wise (layer-wise) gradient normalization for training DNNs to front the gradient explosion or vanishing problem.”
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks (2017, background): “The normalized and de-correlated activation is well known for improving the conditioning of the Fisher information matrix and accelerating the training of deep neural networks (LeCun et al. 1998; Desjardins et al. 2015; Yu et al. 2017).”
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks (2019, unknown): “[36], who used layer-wise gradient normalization to alleviate both vanishing and exploding gradients.”