Combining Linear Regression Models

By Zheng Yuan, Yuhong Yang. Journal of the American Statistical Association, 2005

158 citations recorded by the source corpus.

Model-combining (i.e., mixing) methods have been proposed in recent years to deal with uncertainty in model selection. Even though advantages of model combining over model selection have been demonstrated in simulations and data examples, it is still unclear to a large extent when model combining should be preferred. In this work, first we propose an instability measure to capture the uncertainty of model selection in estimation, called perturbation instability in estimation (PIE), based on perturbation of the sample. We demonstrate that estimators from model selection can have large PIE values and that model combining substantially reduces the instability for such cases. Second, we propose a model combining method, adaptive regression by mixing with model screening (ARMS), and derive a theoretical property. In ARMS, a screening step is taken to narrow down the list of candidate models before combining, which not only saves computing time, but also can improve estimation accuracy. Third, we compare ARMS with EBMA (an empirical Bayesian model averaging) and model selection methods in a number of simulations and real data examples. The comparison shows that model combining produces better estimators when the instability of model selection is high and that ARMS performs better than EBMA in most such cases in our simulations. With respect to the choice between model selection and model combining, we propose a rule of thumb in terms of PIE. The empirical results support that PIE is a sensible indicator of model selection instability in estimation and is useful for understanding whether model combining is a better choice over model selection for the data at hand.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/345d35bb850b721b978d4cb97c6208cead4e8516

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference (2018, methodology): “Note 668 that we do not include adaptive regression by mixing with model screening (ARMS: 669 Yang, 2001) because its more sophisticated variations (Yuan and Yang, 2005) are not 670 readily implemented in R, and the basic ARMS is barely dierent from AIC-model 671 averaging for a preselected set of ”
A Mallows-Type Model Averaging Estimator for the Varying-Coefficient Partially Linear Model (2018, methodology): “For example, model screening could be introduced into the analysis, and judging from existing results (e.g., Yuan and Yang 2005), it is conceivable that removing the poorest models before averaging can contribute to greater estimation and predictive efficiency.”
INFERENCE AFTER MODEL AVERAGING IN LINEAR REGRESSION MODELS (2018, unknown): “…1997; Hjort and Claeskens, 2003a; Zhang and Liang, 2011; Zhang, Wan, and Zhou, 2012), adaptive regression by mixing models (Yang, 2000, 2001; Yuan and Yang, 2005), Mallows’ Cp-type averaging (Hansen, 2007; Wan, Zhang, and Zou, 2010; Liu and Okui, 2013; Zhang, Zou, and Liang, 2014), optimal…”
Model averaging and weight choice in linear mixed-effects models (2014, methodology): “Screening using AIC and BIC has been advocated by Yuan & Yang (2005), and stepwise screening has been used in Claeskens et al. (2006) and Zhang et al. (2012).”
Model averaging based on leave-subject-out cross-validation (2016, methodology): “Another method proposed by Yuan and Yang (2005) is a ‘top m’ model screening with the aid of information criterion (AIC or BIC), i.e., themethod excludes candidatemodels whose information criterion values are larger than the mth smallest one.”
Optimal Model Averaging Estimation for Generalized Linear Models and Generalized Linear Mixed-Effects Models (2016, methodology): “This idea is inspired by the work of Yuan and Yang (2005) and Zhang, Lu, and Zou (2013), where they selectedm candidate models with smallest information criterion values.”
Confidence sets for model selection by F -testing (2015, background): “In the literature of model selection, this issue is sometimes referred to as model selection uncertainty (Chat eld, 1995, Draper, 1995, Hoeting et al., 1999, Yuan and Yang, 2005).”