Benign overfitting in linear regression

By Peter L. Bartlett, Philip M. Long, Gábor Lugosi, Alexander Tsigler. 2019

961 citations recorded by the source corpus.

The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lie in an infinite-dimensional space vs. when the data lie in a finite-dimensional space with dimension that grows faster than the sample size.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Percy Liang
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Quoc V. Le
  5. Ruslan Salakhutdinov
  6. Sergey Levine
  7. Jianfeng Gao
  8. Luke S. Zettlemoyer

In Computational Mechanics

  1. Robert Tibshirani
  2. Percy Liang
  3. Trevor J. Hastie
  4. Francis R. Bach
  5. Vitaly Feldman
  6. Tengyuan Liang
  7. Christopher R'e
  8. Martin Jaggi

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Jitendra Malik
  6. Bernt Schiele
  7. Xiaogang Wang
  8. Andrew Zisserman

In Scaling & Learning Theory

  1. Vitaly Feldman
  2. Percy Liang
  3. Tengyuan Liang
  4. Aditi Raghunathan
  5. Chiyuan Zhang
  6. Andrea Montanari
  7. Taiji Suzuki
  8. Benoit Sanchez

Citations in the corpus