Theoretical Analysis of Auto Rate-Tuning by Batch Normalization

By Sanjeev Arora, Zhiyuan Li, Kaifeng Lyu. 2018

146 citations recorded by the source corpus.

Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates. It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, $0.3$), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of $T^{-1/2}$ in $T$ iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates. A similar result with convergence rate $T^{-1/4}$ is also shown for stochastic gradient descent.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Weight Standardization (2019, background): “[1] provides theoretical analysis on the auto learning rate tuning property of BN.”
An Exponential Learning Rate Schedule for Deep Learning (2019, background): “(Arora et al., 2019) show the gradient is always perpendicular to the current parameter vector which has the effect that norm of each scale invariant parameter group increases monotonically, which has an auto-tuning effect.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “This is mainly due to its scale-invariant property [20], [183], [184], [185], i.e ., it does not change the prediction when rescaling parameters and works by adaptively adjusting the learning rate in a layer-wise manner [92], [94], [160].”
Batch normalization provably avoids ranks collapse for randomly initialised deep networks (2020, background): “…literature: decoupling optimization of direction and length of the parameters [20], auto-tuning of the learning rate for stochastic gradient descent [3], widening the learning rate range [7], alleviating sharpness of the Fisher information matrix [18], and smoothing the optimization landscape [25].”
Positional Normalization (2019, methodology): “…of normalization methods was initially attributed to “reducing internal covariate shift” in hidden layers [29, 41], an array of recent studies [2, 3, 5, 25, 48, 60, 69, 77] has provided evidence that BN changes the loss surface and prevents divergence even with large step sizes [5], which…”
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (2020, background): “Arora et al. [45] argued that batch normalization reduces the importance of tuning the learning rate, while Li and Arora [46] showed that models trained using batch normalization can converge even if the learning rate increases exponentially during training.”