Data-Dependent Path Normalization in Neural Networks

By Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, Nati Srebro. 2015

25 citations recorded by the source corpus.

We propose a unified framework for neural net normalization, regularization and optimization, which includes Path-SGD and Batch-Normalization and interpolates between them across two different dimensions. Through this framework we investigate issue of invariance of the optimization, data dependence and the connection with natural gradients.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Robust Large Margin Deep Neural Networks (2016, background): “in metrics. 5To simplify the derivation we omit the bias vectors and therefore also the centering applied by the batch normalization. This does not affect the generality of the result. We also follow [37] and omit the batch normalization scaling, as it can be included into the weight matrix of the l”
Exploring Generalization in Deep Learning (2017, methodology): “When calculating norms on a network with a Batch Normalization layer, we reparametrize the network to one that represents the exact same function without Batch Normalization as suggested in [20].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “The scale-invariant weight vector in a network is always perpendicular to its gradient [21], [94], [183], [184], [186], which has an auto-tuning effect [91], [94], [187], [188].”
Visualizing Deep Network Training Trajectories with PCA (2016, background): “• Would techniques that normalize the weights change the picture, such as batch normalization (Ioffe & Szegedy, 2015), path normalization (Neyshabur et al., 2016), natural gradient (Amari, 1998), or various kinds of data-dependent initialization (Kraehenbuehl et al., 2016; Mishkin & Matas, 2016)?”
Implicit Regularization in Deep Learning (2017, methodology): “When calculating norms on a network with a Batch Normalization layer, we reparametrize the network to one that represents the exact same function without Batch Normalization as suggested in [60].”
Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation (2019, background): “In contrast, the bounds of Neyshabur et al. (2015); Bartlett et al. (2017); Neyshabur et al. (2017a); Golowich et al. (2017) all depend on a product of norms of weight matrices which scales exponentially in the network depth, and which can be thought of as a worst case Lipschitz constant of the…”
Centered Weight Normalization in Accelerating Training of Deep Neural Networks (2017, methodology): “Indeed, batch normalization is invariant to the weights scaling as analyzed in [24] and [4].”