Towards a Theoretical Understanding of Batch Normalization

By Jonas Köhler, Hadi Daneshmand, Aurélien Lucchi, M. Zhou, Klaus Neymeyr, Thomas Hofmann. 2018

50 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Weight Standardization (2019, background): “[19] explores the effect of length-direction decoupling used in BN and [36].”
How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, background): “Recent work [14] identifies simple, concrete settings where a variant of training with BatchNorm provably improves over standard training algorithms.”
A Mean Field Theory of Batch Normalization (2019, background): “(Kohler et al., 2018) identified situations in which batchnorm can provably induce acceleration in training.”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, methodology): “BN has been extensively used in various network architectures [11, 43, 12, 50, 42, 14] for its benefits in improving both the optimization efficiency [19, 9, 21, 5, 37] and generalization capability [19, 3, 5, 48].”
Information Geometry of Orthogonal Initializations and Training (2018, result): “Furthermore, we observe that the rate at which Stiefel and Oblique networks decrease training loss is not dependent on their gradient smoothness, a result which is consistent with the recent analysis of [20].”
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization (2018, background): “Kohler et al. (2018) aims to ﬁnd theoretical setting such that training neural networks with BatchNorm is faster than without BatchNorm.”
Dropout vs. batch normalization: an empirical study of their impact to deep learning (2020, background): “However, batch normalization also has a regularization effect that renders dropout unnecessary in some cases, as documented in the original paper [7], in [20], and [9].”
DO-Conv: Depthwise Over-Parameterized Convolutional Layer (2020, background): “Normalization layers have been widely used for improving the training of CNNs, while the reason for their success is still being actively studied [31, 4, 21, 20].”
Whitening and Coloring Batch Transform for GANs (2018, background): “In fact, feature normalization makes the loss landscape smoother (Santurkar et al. (2018); Kohler et al. (2018)), making the gradient-descent weight updates closer to Newton updates (Huang et al. (2018); LeCun et al. (2012); S. Wiesler (2011)).”