Path-SGD: Path-Normalized Optimization in Deep Neural Networks

By Behnam Neyshabur, Ruslan Salakhutdinov, Nati Srebro. 2015

339 citations recorded by the source corpus.

We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with respect to a path-wise regularizer related to max-norm regularization. Path-SGD is easy and efficient to implement and leads to empirical gains over SGD and Ada-Grad.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Layer Normalization (2016, methodology): “Re-parameterization in the ReLU network was studied in the Path-normalized SGD [Neyshabur et al., 2015].”
Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks (2019, background): “The implicit regularization provided by the choice of the optimization algorithm has recently become an active area of research in machine learning, putting lot of interest on the behavior of gradient descent on deep over-parametrized models [Neyshabur et al., 2015b, 2017, Zhang et al., 2017].”
FractalNet: Ultra-Deep Neural Networks without Residuals (2016, neutral): “Path-SGD [26] offers an alternative normalization scheme.”
Sharp Minima Can Generalize For Deep Nets (2017, background): “Let us redeﬁne, for convenience, the non-negative homogeneity property (Neyshabur et al., 2015; Lafond et al., 2016) below.”
Robust Large Margin Deep Neural Networks (2016, background): “A modified version of stochastic gradient descent for optimization of DNNs that are invariant to weight rescaling in different layers is proposed in [22], where it is shown that such an optimization may lead to a smaller generalization error (GE)—the difference between the empirical error and the ex”
Regularisation of neural networks by enforcing Lipschitz continuity (2018, methodology): “Neyshabur et al. (2015) presents the Path-SGD optimisation algorithm, a method speciﬁcally designed for training deep networks.”
Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data (2017, background): “(Neyshabur, Salakhutdinov, and Srebro [NSS15] propose training neural networks via steepest descent with respect to the path norm.”
The Marginal Value of Adaptive Gradient Methods in Machine Learning (2017, background): “In a similar vein, Neyshabur, Salakhutdinov, and Srebro [14] show that a diﬀerent algorithm, one which performs descent using a metric that is invariant to re-scaling of the parameters, can lead to solutions which sometimes generalize better than SGD.”
Exploring Generalization in Deep Learning (2017, methodology): “• When training the same architecture, with the same training set, using two different optimization methods (or different algorithmic or parameter choices), one method results in better generalization even though both lead to zero training error [18, 12].”
The Implicit Bias of Gradient Descent on Separable Data (2017, background): “…increasingly clear that implicit biases introduced by the optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models (Neyshabur et al., 2014, 2015; Zhang et al., 2017; Keskar et al., 2017; Neyshabur et al., 2017; Wilson et al., 2017).”
Stronger generalization bounds for deep nets via a compression approach (2018, methodology): “This appeared in Neyshabur et al. [2017b] that showed for one layer we can even use ‖η‖ 1.”
Generalization in Deep Learning (2017, background): “…spaces F , there are several well-known bounds on R m ( F ) including those with explicit exponential dependence on depth (Sun et al., 2016; Neyshabur et al., 2015b; Xie et al., 2015) and explicit linear dependence on the number of trainable parameters (Shalev-Shwartz and Ben-David, 2014).”
Fantastic Generalization Measures and Where to Find Them (2019, background): “…measures that we consider in this work belong to a few different families: PAC-Bayes (McAllester, 1999; Dziugaite and Roy, 2017; Neyshabur et al., 2017); VC-dimension (Vapnik and Chervonenkis, 1971); and norm-based bounds (Neyshabur et al., 2015b; Bartlett et al., 2017; Neyshabur et al., 2018a).”
Emergence of Invariance and Disentanglement in Deep Representations (2017, background): “eem to induce overtting. Consequently, a number of alternative measures of complexity have been proposed that capture the intuitive biasvariance trade-o curve, such as dierent norms of the weights (Neyshabur et al., 2015). From the discussion above, we have seen that the quantity of information in t”
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks (2018, background): “It is interesting to understand the implicit regularization effects of the optimization algorithms [19, 8, 25] for neural networks, which we leave for future work.”
Implicit Bias of Gradient Descent on Linear Convolutional Networks (2018, methodology): “Implicit biases introduced by optimization algorithms play an crucial role in learning deep neural networks [Neyshabur et al., 2015b,a, Hochreiter and Schmidhuber, 1997, Keskar et al., 2016, Chaudhari et al., 2016, Dinh et al., 2017, Andrychowicz et al., 2016, Neyshabur et al., 2017, Zhang et al.,…”
Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima (2017, background): “As reported by (Neyshabur et al., 2015), it is desirable to have scaling-invariant learning algorithm to stabilize the training process.”
The Singular Values of Convolutional Layers (2018, neutral): “, 2018)2 and path-norm regularizers (Neyshabur et al., 2015).”
Shampoo: Preconditioned Stochastic Tensor Optimization (2018, methodology): “Whereas recent optimization methods, such as (Martens and Grosse, 2015; Neyshabur et al., 2015), need to be aware of the structure of the underlying model, Shampoo only needs to be informed of the tensors involved and their sizes.”