Minnorm training: an algorithm for training over-parameterized deep neural networks

By Yamini Bansal, Madhu S. Advani, David D. Cox, Andrew M. Saxe. 2018

18 citations recorded by the source corpus.

In this work, we propose a new training method for finding minimum weight norm solutions in over-parameterized neural networks (NNs). This method seeks to improve training speed and generalization performance by framing NN training as a constrained optimization problem wherein the sum of the norm of the weights in each layer of the network is minimized, under the constraint of exactly fitting training data. It draws inspiration from support vector machines (SVMs), which are able to generalize well, despite often having an infinite number of free parameters in their primal form, and from recent theoretical generalization bounds on NNs which suggest that lower norm solutions generalize better. To solve this constrained optimization problem, our method employs Lagrange multipliers that act as integrators of error over training and identify `support vector'-like examples. The method can be implemented as a wrapper around gradient based methods and uses standard back-propagation of gradients from the NN for both regression and classification versions of the algorithm. We provide theoretical justifications for the effectiveness of this algorithm in comparison to early stopping and $L_2$-regularization using simple, analytically tractable settings. In particular, we show faster convergence to the max-margin hyperplane in a shallow network (compared to vanilla gradient descent); faster convergence to the minimum-norm solution in a linear chain (compared to $L_2$-regularization); and initialization-independent generalization performance in a deep linear network. Finally, using the MNIST dataset, we demonstrate that this algorithm can boost test accuracy and identify difficult examples in real-world datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, neutral): “is correctly classified) [24, 25, 26, 27].”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Beyond the jamming point, the accuracy keeps steadily improving as the number of parameters increases [25, 26, 27], although it does so quite slowly.”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, background): “A surprising observation is that their performance on supervised tasks keeps increasing with their width h in the over-parametrized regime where they already ﬁt all the training data [Neyshabur et al., 2017, Bansal et al., 2018, Advani and Saxe, 2017, Spigler et al., 2018].”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “2B-C), and beyond this point the accuracy keeps improving as the number of parameters increases [32, 31, 4], although it does so quite slowly.”
Geometric compression of invariant manifolds in neural nets (2020, background): “For a large anisotropy, the effective dimension of the data is reduced, improving kernel 1MNIST or CIFAR present an effective dimension dM ∈ [15, 35] [3]and p ≈ 6 · 104.”
Direct-fit to nature: an evolutionary perspective on biological (and artificial) neural networks (2019, background): “(D) Complex models such as ANNs, however, can nonetheless yield a fit that both captures the training data and generalizes well to novel data within the scope of the training sample (see Panel G and Bansal et al., 2018, for a related discussion).”