Classification vs regression in overparameterized regimes: Does the loss function matter?

By Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel J. Hsu, Anant Sahai. 2020

172 citations recorded by the source corpus.

We compare classification and regression tasks in the overparameterized linear model with Gaussian features. On the one hand, we show that with sufficient overparameterization all training points are support vectors: solutions obtained by least-squares minimum-norm interpolation, typically used for regression, are identical to those produced by the hard-margin support vector machine (SVM) that minimizes the hinge loss, typically used for training classifiers. On the other hand, we show that there exist regimes where these solutions are near-optimal when evaluated by the 0-1 test loss function, but do not generalize if evaluated by the square loss function, i.e. they achieve the null risk. Our results demonstrate the very different roles and properties of loss functions used at the training phase (optimization) and the testing phase (generalization).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

Citations in the corpus

Mixed-Privacy Forgetting in Deep Networks (2020, background): “[34] Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai.”
Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks (2020, background): “See (Muthukumar et al., 2020) for a more in-depth theoretical discussion of loss functions and the related literature.”
Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019, background): “Recent studies of high-dimensional linear discriminant analysis [15, 51, 66] share important features with the model studied here: anisotropic covariance and signal aligned with its top eigenvectors.”
Implicit Regularization via Neural Feature Alignment (2020, background): “Recently, new analysis of minimum norm interpolators and max margin solutions for overparametrized linear models emphasize the key role of feature geometry, and speciﬁcally feature anisotropy, in the generalization performance (Bartlett et al., 2019; Muthukumar et al., 2019, 2020; Xie et al., 2020).”