Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss

By Lénaïc Chizat, Francis R. Bach. 2020

387 citations recorded by the source corpus.

Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. Towards understanding this phenomenon, we analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations. We show that the limits of the gradient flow on exponentially tailed losses can be fully characterized as a max-margin classifier in a certain non-Hilbertian space of functions. In presence of hidden low-dimensional structures, the resulting margin is independent of the ambiant dimension, which leads to strong generalization bounds. In contrast, training only the output layer implicitly solves a kernel support vector machine, which a priori does not enjoy such an adaptivity. Our analysis of training is non-quantitative in terms of running time but we prove computational guarantees in simplified settings by showing equivalences with online mirror descent. Finally, numerical experiments suggest that our analysis describes well the practical behavior of two-layer neural networks with ReLU activation and confirm the statistical benefits of this implicit bias.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, methodology): “…has also been studied in various settings, suggesting a bias towards large-margin, low-norm, or low-rank solutions [Gunasekar et al., 2017, 2018, Soudry et al., 2018, Li et al., 2018, Ji and Telgarsky, 2019a,b, Arora et al., 2019a, Lyu and Li, 2019, Chizat and Bach, 2020, Razin and Cohen, 2020].”
Feature Learning in Infinite-Width Neural Networks (2020, background): “A notable exception is [10], which showed that the mean field limit, but not the NTK limit, can learn low dimension linear structure of the input distribution resulting in ambient-dimension-independent generalization bounds.”
When do neural networks outperform kernel methods? (2020, background): “Further even in the infinite width limit, it is known that two-layers neural networks can actually capture a richer class of functions than the associated RKHS, provided SGD training is scaled differently from the lazy regime [23, 7, 28, 30, 8].”
Geometric compression of invariant manifolds in neural nets (2020, background): “For gradient descent, with the logistic loss for a one-hidden layer can be shown to correspond to a max-margin classifier in a certain non-Hilbertian space of functions [32].”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “Second, its applicability to state-of-the-art neural networks is limited by the kernel regime of networks, which does not capture many interesting and useful deep learning phenomena 62,78 .”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “Works along this line have treated: linear (single-layer) predictors ([67, 31, 40, 2]); normalized linear models ([71]); certain polynomially parameterized linear models ([70]); homogeneous (and sum of homogeneous) models ([53, 49]); ultra-wide neural networks ([38, 51, 14, 58]); linear neural netwo”