Surprises in High-Dimensional Ridgeless Least Squares Interpolation

By Trevor J. Hastie, Andrea Montanari, Saharon Rosset, Robert Tibshirani. 2019

892 citations recorded by the source corpus.

Interpolators—estimators that achieve zero training error—have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum ℓ2 norm (“ridgeless”) interpolation least squares regression, focusing on the high-dimensional regime in which the number of unknown parameters p is of the same order as the number of samples n. We consider two different models for the feature distribution: a linear model, where the feature vectors xi∈ℝp are obtained by applying a linear transform to a vector of i.i.d. entries, xi = Σ1/2zi (with zi∈ℝp); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, xi = φ(Wzi) (with zi∈ℝd, W∈ℝp×d a matrix of i.i.d. entries, and φ an activation function acting componentwise on Wzi). We recover—in a precise quantitative way—several phenomena that have been observed in large-scale neural networks and kernel machines, including the “double descent” behavior of the prediction risk, and the potential benefits of overparametrization.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Mathematical Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

  1. Robert Tibshirani
  2. Percy Liang
  3. John C. Duchi
  4. Trevor J. Hastie
  5. Tengyu Ma
  6. Christopher R'e
  7. Florent Krzakala
  8. Lenka Zdeborov'a

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Jitendra Malik
  5. Ming-Hsuan Yang
  6. Xiaogang Wang
  7. Andrew Zisserman
  8. Bernt Schiele

In Mathematical Physics

  1. Vitaly Feldman
  2. E.J. Candes
  3. M. Soltanolkotabi
  4. Percy Liang
  5. Lucas K. Mentch
  6. Xialiang Dou
  7. Siyu Zhou
  8. Tengyuan Liang

In Scaling & Learning Theory

  1. Matthieu Wyart
  2. Mario Geiger
  3. Stefano Spigler
  4. Vitaly Feldman
  5. Tengyu Ma
  6. Arthur Jacot
  7. Greg Yang
  8. Jaehoon Lee

Citations in the corpus