What You See May Not Be What You Get: A Brief, Nontechnical Introduction to Overfitting in Regression-Type Models

By M. Babyak. Psychosomatic Medicine, 2004

2,235 citations recorded by the source corpus.

Objective: Statistical models, such as linear or logistic regression or survival analysis, are frequently used as a means to answer scientific questions in psychosomatic research. Many who use these techniques, however, apparently fail to appreciate fully the problem of overfitting, ie, capitalizing on the idiosyncrasies of the sample at hand. Overfitted models will fail to replicate in future samples, thus creating considerable uncertainty about the scientific merit of the finding. The present article is a nontechnical discussion of the concept of overfitting and is intended to be accessible to readers with varying levels of statistical expertise. The notion of overfitting is presented in terms of asking too much from the available data. Given a certain number of observations in a data set, there is an upper limit to the complexity of the model that can be derived with any acceptable degree of uncertainty. Complexity arises as a function of the number of degrees of freedom expended (the number of predictors including complex terms such as interactions and nonlinear terms) against the same data set during any stage of the data analysis. Theoretical and empirical evidence—with a special focus on the results of computer simulation studies—is presented to demonstrate the practical consequences of overfitting with respect to scientific inference. Three common practices—automated variable selection, pretesting of candidate predictors, and dichotomization of continuous variables—are shown to pose a considerable risk for spurious findings in models. The dilemma between overfitting and exploring candidate confounders is also discussed. Alternative means of guarding against overfitting are discussed, including variable aggregation and the fixing of coefficients a priori. Techniques that account and correct for complexity, including shrinkage and penalization, also are introduced.

Subfields: Computational Mechanics, Computer Networks and Communications, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/989709b927a3e485ec8ecd12955ddbb854133902

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Networks and Communications

In Statistics and Probability

Citations in the corpus

The number of subjects per variable required in linear regression analyses. (2015, background): “Furthermore, the estimated R(2) statistic of a linear regression model can be artificially inflated as the number of subjects per variable (SPV) decreases [when outcomes are binary or time-to-event in nature, the corresponding quantities are the effective sample size, denoting the number of observed”
Teacher Well-Being and the Implementation of School-Wide Positive Behavior Interventions and Supports (2012, result): “With a sample of only 184 teachers, the problem of overfitting was a concern, and with that in mind, the analysis still may have led to some overfitting and spurious results (Babyak, 2004; Burnham & Anderson, 2002 ), which may be further exaggerated in the multilevel model (Snijders & Bosker, 1993).”
Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation. (2017, background): “known in many disciplines as overfitting (38, 338, 345) or the curse of dimensionality (821).”
Why do we respond so differently? Reviewing determinants of human salivary cortisol responses to challenge. (2009, background): “However, researchers should be aware of the fact that the appropriate number of covariates depends on the sample size since model overfitting might lead to spurious results (Babyak, 2004).”
Comparison of different regression models and validation techniques for the assessment of wheat leaf area index from hyperspectral data (2015, background): “This phenomenon can be defined as an error that occurs in regression modelling when a model is too closely fit to a limited number of samples (Babyak 2004; Hawkins 2004).”
Moving Forward: Challenges and Directions for Psychopathological Network Theory and Methodology (2017, background): “However, this complicated model describes not only the relationship between depression and neuroticism but also measurement error, which leads to overfitting (Babyak, 2004).”
Distress in couples coping with cancer: a meta-analysis and critical review of role and gender effects. (2008, methodology): “Introduction of control variables or testing the simplest of mediational or moderator variables would involve overfitted regression equations or a pre-selection of variables based on preliminary analyses that capitalized on chance, with the net result that findings would not be generalizable (Babyak”
Stability and variability of personality networks. A tutorial on recent developments in network psychometrics (2017, methodology): “…matrices and because in partial correlation networks an increase in the number of nodes can lead to overfitting and very unstable estimates (Babyak, 2004), such networks are usually estimated using regularization methods such as the least absolute shrinkage and selection operator (lasso;…”
The Application of Multiple Linear Regression and Artificial Neural Network Models for Yield Prediction of Very Early Potato Cultivars before Harvest (2021, background): “the number of observations made, the model’s ability to detect real relationships is severely limited [49].”
Using Lasso for Predictor Selection and to Assuage Overfitting: A Method Long Overlooked in Behavioral Sciences (2015, methodology): “Although the model R2 values are not vastly different as would be anticipated by a simulation by Babyak (2004) because of the large n to p ratio, the R2 for Lasso is slightly smaller (79% vs.”
A global predictive model of carbon in mangrove soils (2014, methodology): “Constructing a robust predictive model requires balancing the ability to explain the most variation in the data sample with the model’s generalizability, i.e. its ability to predict out of sample (Babyak 2004).”