A LASSO FOR HIERARCHICAL INTERACTIONS

By Jacob Bien, Jonathan E. Taylor, Robert Tibshirani. 2012

523 citations recorded by the source corpus.

We add a set of convex constraints to the lasso to produce sparse interaction models that honor the hierarchy restriction that an interaction only be included in a model if one or both variables are marginally important. We give a precise characterization of the effect of this hierarchy constraint, prove that hierarchy holds with probability one and derive an unbiased estimate for the degrees of freedom of our estimator. A bound on this estimate reveals the amount of fitting “saved” by the hierarchy constraint. We distinguish between parameter sparsity—the number of nonzero coefficients—and practical sparsity—the number of raw variables one must measure to make a new prediction. Hierarchy focuses on the latter, which is more closely tied to important data collection concerns such as cost, time and effort. We develop an algorithm, available in the R package hierNet, and perform an empirical study of our method.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Supersparse linear integer models for optimized medical scoring systems (2015, positive): “This presents a practical alternative to create classification models that obey structured sparsity constraints (see e.g., Jenatton et al., 2011) or hierarchical constraints (see e.g., Bien et al., 2013).”
A significance test for forward stepwise model selection (2014, methodology): “The method imposes a strong hierarchical constraint on interactions (similar to that in Bien et al. (2012)) where an interaction term can only be included if both its main effects are also included.”
Proximal Algorithms in Statistics and Machine Learning (2015, background): “Although such approaches are commonplace in statistics and machine learning [Bien et al., 2013], there hasn’t been a real focus on the general family of approaches that underly these algorithms.”
Cross-Validation, Shrinkage and Variable Selection in Linear Regression Revisited (2013, background): “In a related context [22] refer to parameter sparsity and practical sparsty.”
Identifying a Minimal Class of Models for High-dimensional Data (2015, methodology): “, 2010) or by solving a relevant optimization problem (Bien et al., 2013).”
Identifying Significant Predictive Bias in Classifiers (2016, background): “Group lasso or hierarchical lasso [4,22] can adjust for this, but require ex ante speciﬁcation.”
The Kernel Interaction Trick: Fast Bayesian Discovery of Pairwise Interactions in High Dimensions (2019, methodology): “However, these methods do not address how to construct valid confidence intervals or adjust for multiple comparisons1 (Bien et al., 2013; Lim & Hastie, 2015; Nakagawa et al., 2016; Shah, 2016; Wu et al., 2009).”
Multi-Task Feature Interaction Learning (2016, background): “To overcome the dimensionality issues introduced by interaction effects, two types of heredity constraints have been studied [5]; namely strong hierarchy in which an interaction effect can be selected into the model only if both of its corresponding linear effects have been selected, and weak hierar”
Identifying key variables and interactions in statistical models of building energy consumption using regularization (2015, background): “[47] introduce constraints that preserve strong orweak hierarchy between interactions.”
Group Regularized Estimation Under Structural Hierarchy (2014, background): “The past works in this direction include SHIM (Choi, Li, and Zhu 2010), VANISH (Radchenko and James 2010), and HL (Bien, Taylor, and Tibshirani 2013).”
Learning Sparse Additive Models with Interactions in High Dimensions (2016, background): “Within this setting, there exist algorithms that recover S1,S2, along with convergence rates for estimating f , but only in the limit of large n [30, 15, 31].”
Short term electricity demand forecasting using partially linear additive quantile regression with an application to the unit commitment problem (2018, methodology): “The model is given as [3]: yth = β0h,τ + ∑ i βhi,τzthi + ∑ i 6=k αikzthizthk + εth,τ .”
Hierarchical Shrinkage Priors for Regression Models (2017, methodology): “The performance of the hierarchical prior introduced in this paper was compared to the hierarchical lasso (Bien et al., 2013) and three spike-and-slab priors for variable selection for interactions (Chipman, 1996) using five-fold cross-validation.”
Exact Post-Selection Inference for Sequential Regression Procedures (2014, methodology): “Aside from the lasso, examples in this problem form include the generalized lasso (Tibshirani & Taylor 2011), and hierNet (Bien et al. 2013).”
Performance-influence models for highly configurable systems (2015, methodology): “Thus, based on our experience and following standard practices of machine learning [2], we perform our learning hierarchically: We first start adding only the individual option influences and then add interactions as candidates containing options that have been found already to contribute to perform”