Stepwise Model Fitting and Statistical Inference: Turning Noise into Signal Pollution

By Roger Mundry, Charles L. Nunn. American Naturalist, 2008

438 citations recorded by the source corpus.

Statistical inference based on stepwise model selection is applied regularly in ecological, evolutionary, and behavioral research. In addition to fundamental shortcomings with regard to finding the “best” model, stepwise procedures are known to suffer from a multiple‐testing problem, yet the method is still widely used. As an illustration of this problem, we present results of a simulation study of artificial data sets of uncorrelated variables, with two to 10 predictor variables and one dependent variable. We then compared results from stepwise regression with a regression model in which all predictor variables were entered simultaneously. These analyses clearly demonstrate that significance tests based on stepwise procedures lead to greatly inflated Type I error rates (i.e., the probability of erroneously rejecting a true null hypothesis). By using a simple simulation design, our study amplifies previous warnings about using stepwise procedures, and we follow others in recommending that biologists refrain from applying these methods.

Subfields: Artificial Intelligence, Ecological Modeling, Nature and Landscape Conservation

View Paper: https://www.semanticscholar.org/paper/01c035525d779f271664b90e3a679cfdca90f1a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Ecological Modeling

In Nature and Landscape Conservation

Citations in the corpus

Changing philosophies and tools for statistical inferences in behavioral ecology (2009, background): “Behavioral Ecology doi:10.1093/beheco/arp137 Advance Access publication 2 November 2009 Review Changing philosophies and tools for statistical inferences in behavioral ecology”
Issues in information theory-based statistical inference—a commentary from a frequentist’s perspective (2010, methodology): “Behav Ecol Sociobiol (2011) 65:57–68 61 to that used by Mundry and Nunn (2009) .”
Personality and parasites: sex-dependent associations between avian malaria infection and multiple behavioural traits (2011, methodology): “Although model simplification by stepwise deletion has been criticised in the literature (Whittingham et al. 2006; Mundry and Nunn 2009), a recent study validated stepwise deletion as a method of model selection and established that it performed just as well as other methods of producing predictive ”
AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons (2011, methodology): “” The analyst does not even know the second best model when using one of the test-based subset regression methods (Mundry and Numm 2009).”
More than friends? Behavioural and genetic aspects of heterosexual associations in wild chacma baboons (2010, methodology): “The significance of variables was tested using the full models to avoid problems associated with stepwise model selection procedures, such as biased parameter estimation and risks of false positives (Mundry and Nunn 2009).”
Mate competition, testosterone and intersexual relationships in bonobos, Pan paniscus (2012, methodology): “We explicitly decided against using a stepwise approach because of the many shortcomings this procedure has (Whittingham et al. 2006; Mundry & Nunn 2009).”
MMI: Multimodel inference or models with management implications? (2015, background): “When models are too finely tuned to specific aspects of the data in hand, we say the model is overfit, in the sense that the model is accommodating, not only the actual effects of the explanatory variables, but also the noise inherent in any real data set (Mundry and Nunn 2009).”
Information-theoretic approaches to statistical analysis in behavioural ecology: an introduction (2010, methodology): “In cases when authors do apply model selection methods, these are usually based on stepwise or stepdown techniques, which follow a strategy that is philosophically different from that of the IT method (Draper and Smith 1981: 327–347; Derksen and Keselman 1992; Whittingham et al. 2006; Mundry and Nun”