Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolation

By Partha P. Mitra. 2019

50 citations recorded by the source corpus.

Traditionally in regression one minimizes the number of fitting parameters or uses smoothing/regularization to trade training (TE) and generalization error (GE). Driving TE to zero by increasing fitting degrees of freedom (dof) is expected to increase GE. However modern big-data approaches, including deep nets, seem to over-parametrize and send TE to zero (data interpolation) without impacting GE. Overparametrization has the benefit that global minima of the empirical loss function proliferate and become easier to find. These phenomena have drawn theoretical attention. Regression and classification algorithms have been shown that interpolate data but also generalize optimally. An interesting related phenomenon has been noted: the existence of non-monotonic risk curves, with a peak in GE with increasing dof. It was suggested that this peak separates a classical regime from a modern regime where over-parametrization improves performance. Similar over-fitting peaks were reported previously (statistical physics approach to learning) and attributed to increased fitting model flexibility. We introduce a generative and fitting model pair ("Misparametrized Sparse Regression" or MiSpaR) and show that the overfitting peak can be dissociated from the point at which the fitting function gains enough dof's to match the data generative model and thus provides good generalization. This complicates the interpretation of overfitting peaks as separating a "classical" from a "modern" regime. Data interpolation itself cannot guarantee good generalization: we need to study the interpolation with different penalty terms. We present analytical formulae for GE curves for MiSpaR with $l_2$ and $l_1$ penalties, in the interpolating limit $\lambda\rightarrow 0$.These risk curves exhibit important differences and help elucidate the underlying phenomena.

Subfields: Applied Mathematics, Computational Mechanics, Mathematical Physics, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Computational Mechanics

In Mathematical Physics

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Deep double descent: where bigger models and more data hurt (2019, methodology): “The PyTorch Paszke et al (2017) specification of our ResNets and CNNs are available at https://gitlab.com/harvard-machine-learning/double-descent/tree/master. ResNets. We define a family of ResNet18s of increasing size as follows. We follow the Preactivation ResNet18 architecture of He et al (2016),”
Classification vs regression in overparameterized regimes: Does the loss function matter? (2020, background): “Subsequent work [4, 8, 22, 26, 29, 27] has identified theoretical conditions under which overparameterization and interpolation can be helpful, or at the very least, harmless in linear regression with different feature families.”
Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model (2021, background): “Other examples include models of the double descent phenomenon [40].”