Forest Garrote

By Nicolai Meinshausen. 2009

18 citations recorded by the source corpus.

Variable selection for high-dimensional linear models has received a lot of attention lately, mostly in the context of l1-regularization. Part of the attraction is the variable selection effect: parsimonious models are obtained, which are very suitable for interpretation. In terms of predictive power, however, these regularized linear models are often slightly inferior to machine learning procedures like tree ensembles. Tree ensembles, on the other hand, lack usually a formal way of variable selection and are difficult to visualize. A Garrote-style convex penalty for trees ensembles, in particular Random Forests, is proposed. The penalty selects functional groups of nodes in the trees. These could be as simple as monotone functions of individual predictor variables. This yields a parsimonious function fit, which lends itself easily to visualization and interpretation. The predictive power is maintained at least at the same level as the original tree ensemble. A key feature of the method is that, once a tree ensemble is fitted, no further tuning parameter needs to be selected. The empirical performance is demonstrated on a wide array of datasets.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Iterative random forests to discover predictive and stable high-order interactions (2017, methodology): “Popular statistical and machine-learning methods for detecting interactions among features include decision trees and their ensembles: CART (12), random forests (RFs) (13), Node Harvest (14), Forest Garrote (15), and Rulefit3 (16), as well as methods more specific to gene–gene interactions with cate”
Node harvest: simple and interpretable regression and classication (2009, background): “In terms of predictive power, rule ensembles seem to be often better than NH and also Random Forests in our experience if the signal-to-noise is high [Meinshausen (2009)].”
A random forest guided tour (2015, methodology): “…suggest aGuided RRF (GRRF), inwhich the importance scores from an ordinary random forest are used to guide the feature selection process in RRF. Lastly, a Garrote-style convex penalty, proposed by Meinshausen (2009), selects functional groups of nodes in trees, yielding to parsimonious estimates.”