Bayesian variable selection with related predictors

By Hugh A. Chipman. 1995

In data sets with many predictors, algorithms for identifying a good subset of predictors are often used. Most such algorithms do not allow for any relationships between predictors. For example, stepwise regression might select a model containing an interaction AB but neither main effect A or B. This paper develops mathematical representations of this and other relations between predictors, which may then be incorporated in a model selection procedure. A Bayesian approach that goes beyond the standard independence prior for variable selection is adopted, and preference for certain models is interpreted as prior information. Priors relevant to arbitrary interactions and polynomials, dummy variables for categorical factors, competing predictors, and restrictions on the size of the models are developed. Since the relations developed are for priors, they may be incorporated in any Bayesian variable selection algorithm for any type of linear model. The application of the methods here is illustrated via the stochastic search variable selection algorithm of George and McCulloch (1993), which is modified to utilize the new priors. The performance of the approach is illustrated with two constructed examples and a computer performance dataset.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Weight Uncertainty in Neural Networks (2015, methodology): “Our prior resembles a spike-and-slab prior (Mitchell and Beauchamp, 1988; George and McCulloch, 1993; Chipman, 1996), where instead all the prior parameters are shared among all the weights.”
The Practical Implementation of Bayesian Model Selection (2001, methodology): “Independence and exchangeable priors on γ may be less satisfactory when the models under consideration contain dependent components such as might occur with interactions , polynomials, lagged variables or indicator variables (Chipman 1996).”
Bayesian Wavelet Regression on Curves With Application to a Spectroscopic Calibration Problem (2001, background): “Carlin and Chib (1995), Chipman (1996), and Geweke (1996), among others, concentrated on special features of these priors.”
Group Regularized Estimation Under Structural Hierarchy (2014, background): “In our setting, there are two types of hierarchy (Chipman, 1996; Bien et al., 2013).”
A Default Bayesian Hypothesis Test for ANOVA Designs (2012, methodology): “We focus on prior distributions for variable selection in regression as this framework provides the basis for the analysis of ANOVA designs (for more information on Bayesian variable selection see Leamer 1978; Zellner 1986, 1987; Mitchell and Beauchamp 1988; Chipman 1996; George and McCulloch 1997).”
Benchmark Priors for Bayesian Model Averaging (2001, background): “regressors. Chipman (1996) examines prior structures that can be used to accommodate general relations between regressors.”
A LASSO FOR HIERARCHICAL INTERACTIONS (2012, methodology): “(2010)]; Bayesian approaches, which specify the hierarchy requirement through a prior [Chipman (1996)]; and, most related to this paper’s proposal, regularized regression methods, which are defined by an optimization problem [Yuan, Joseph and Zou (2009), Zhao, Rocha and Yu (2009), Choi, Li and Zhu (”
Structured variable selection and estimation (2009, background): “Later, Chipman (1996) and Chipman, Hamada and Wu (1997) discussed how the effect heredity can be accommodated in the stochastic search variable selection method developed by George and McCulloch (1993).”
Model uncertainty in cross-country growth regressions (2001, background): “See Chipman (1996) for priors that allow for dependence between regressors.”
The Kernel Interaction Trick: Fast Bayesian Discovery of Pairwise Interactions in High Dimensions (2019, methodology): “Going forward, we model ✓ as being drawn from a Gaussian scale mixture prior to encode desirable properties such as sparsity and strong hierarchy (cf. Carvalho et al., 2009; Chipman, 1996; George & McCulloch, 1993; Griffin & Brown, 2017; Piironen & Vehtari, 2017).”
Projective three-level main effects designs robust to model uncertainty (2000, methodology): “Secondly, a stepwise regression procedure is used for selecting a suitable fitted model, obeying functional marginality (McCullagh & Nelder, 1989, Ch. 3), called strong heredity by Chipman (1996), for the interpretation of the data.”
Weight Uncertainty in Neural Network (2015, methodology): “Our prior resembles a spike-and-slab prior (Mitchell and Beauchamp, 1988; George and McCulloch, 1993; Chipman, 1996), where instead all the prior parameters are shared among all the weights.”