Bayesian regression tree ensembles that adapt to smoothness and sparsity

By Antonio R. Linero, Yun Yang. 2017

187 citations recorded by the source corpus.

Ensembles of decision trees are a useful tool for obtaining flexible estimates of regression functions. Examples of these methods include gradient‐boosted decision trees, random forests and Bayesian classification and regression trees. Two potential shortcomings of tree ensembles are their lack of smoothness and their vulnerability to the curse of dimensionality. We show that these issues can be overcome by instead considering sparsity inducing soft decision trees in which the decisions are treated as probabilistic. We implement this in the context of the Bayesian additive regression trees framework and illustrate its promising performance through testing on benchmark data sets. We provide strong theoretical support for our methodology by showing that the posterior distribution concentrates at the minimax rate (up to a logarithmic factor) for sparse functions and functions with additive structures in the high dimensional regime where the dimensionality of the covariate space is allowed to grow nearly exponentially in the sample size. Our method also adapts to the unknown smoothness and sparsity levels, and can be implemented by making minimal modifications to existing Bayesian additive regression tree algorithms.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Local Linear Forests (2018, background): “Linero and Yang [2018] propose a Bayesian regression tree ensemble tailored to learning smooth, sparse signals and prove posterior minimaxity under certain conditions, highlighting the promise of tree-based methods that can adapt to smoothness.”
DART: Data Addition and Removal Trees (2020, unknown): “…2001) are popular and widely used machine learning models (Lundberg, Erion, and Lee 2018), mainly due to their predictive prowess on many classiﬁcation and regression tasks (Biau, Scornet, and Welbl 2019; Kocev, Vens et al. 2013; Genuer et al. 2017; Wager and Athey 2018; Linero and Yang 2018).”
Machine Unlearning for Random Forests (2020, methodology): “…et al., 1984; Friedman, 2001) are popular and widely used machine learning models (Lundberg et al., 2018), mainly due to their predictive prowess on many classiﬁcation and regression tasks (Kocev et al., 2013; Genuer et al., 2017; Wager & Athey, 2018; Linero & Yang, 2018; Biau et al., 2019).”
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence (2019, background): “2016), and high-dimensional modeling (Linero 2018; Linero and Yang 2018).”
Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects (with Discussion) (2020, unknown): “• Soft-BART (Linero and Yang, 2018).”
Bayesian Nonparametric Causal Inference: Information Rates and Learning Algorithms (2017, methodology): “As it is the case in standard nonparametric regression, the optimal information rate for causal inference gets exponentially slower as we add more dimensions to the feature space [23], [26], [35].”
Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects (2017, unknown): “There have been recent developments on posterior consistency and rates of posterior concentration for Bayesian tree models in prediction contexts (Linero and Yang, 2017; Rockova and van der Pas, 2017).”