Consistent Individualized Feature Attribution for Tree Ensembles

By Scott M. Lundberg, Gabriel Erion-Barner, Su-In Lee. 2018

1,890 citations recorded by the source corpus.

Interpreting predictions from tree ensemble methods such as gradient boosting machines and random forests is important, yet feature attribution for trees is often heuristic and not individualized for each prediction. Here we show that popular feature attribution methods are inconsistent, meaning they can lower a feature's assigned importance when the true impact of that feature actually increases. This is a fundamental problem that casts doubt on any comparison between features. To address it we turn to recent applications of game theory and develop fast exact tree solutions for SHAP (SHapley Additive exPlanation) values, which are the unique consistent and locally accurate attribution values. We then extend SHAP values to interaction effects and define SHAP interaction values. We propose a rich visualization of individualized feature attributions that improves over classic attribution summaries and partial dependence plots, and a unique "supervised" clustering (clustering based on feature attributions). We demonstrate better agreement with human intuition through a user study, exponential improvements in run time, improved clustering performance, and better identification of influential features. An implementation of our algorithm has also been merged into XGBoost and LightGBM, see this http URL for details.

Subfields: Artificial Intelligence, Nature and Landscape Conservation, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Nature and Landscape Conservation

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “…and multiple classiﬁer systems in addition to Deep Learning models; the combination of explanation maps of multiple clas-siﬁers to produce improved explanations of the ensemble to which they belong [243] ; and recent insights dealing with traditional and gradient boosting ensembles [287,288] .”
On the Art and Science of Machine Learning Explanations (2018, methodology): “Both tree shap and a related technique known as treeinterpreter rely instead on traversing internal tree structures to estimate the impact of each x j for some g ( x ) of interest [19], [25].”
Visualizing the Feature Importance for Black Box Models (2018, methodology): “Methods for feature eﬀects include partial dependence (PD) plots [10], individual conditional expectation (ICE) plots [11] and, more recently, SHAP values [15].”
Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning (2021, background): “Representative works include SHAP (Lundberg & Lee, 2017) and its variants (Lundberg et al., 2018; Shrikumar et al., 2017; Schwab & Karlen, 2019), which assign contributions to each feature used in a model, and the sum of the contributions is the difference between a model’s output and the target…”
Mathematical optimization in classification and regression trees (2021, positive): “…number of shortcomings to this approach around, for instance, the generation of data or the loss of accuracy, which have been addressed with enhancements (Botari et al. 2020; Visani et al. 2020; Zhang et al. 2019), or with alternative approaches (Lundberg et al. 2020, 2018; Lundberg and Lee 2017).”
Do Not Trust Additive Explanations (2019, background): “There is a similar idea of calculating differences between the sum of independent effects of variables and joint effect to calculate SHAP interaction values (Lundberg et al., 2018).”
Learning Explainable Models Using Attribution Priors (2019, methodology): “During ﬁne-tuning, we use the same training procedure outlined by Silberman and Guadarrama [19].”
Data Shapley: Equitable Valuation of Data for Machine Learning (2019, background): “Shapley value has been recently proposed as a feature importance score for the purpose of interpreting black-box predictive models (Kononenko et al., 2010; Datta et al., 2016; Lundberg & Lee, 2017; Cohen et al., 2007; Chen et al., 2018; Lundberg et al., 2018).”
Explainable AI Methods - A Brief Overview (2020, background): “Conditional versions exist [50] (for tree-based models only), but the interpretation changes which is a common pitfall [57].”
Scalable and Generalizable Social Bot Detection through Data Selection (2019, methodology): “With only 20 features in our best model ( M196 ), it is possible to interpret its logic using the SHAP model explanation technique (Lundberg, Erion, and Lee 2018).”
DART: Data Addition and Removal Trees (2020, neutral): “Decision trees and random forests (Breiman et al. 1984; Friedman 2001) are popular and widely used machine learning models (Lundberg, Erion, and Lee 2018), mainly due to their predictive prowess on many classiﬁcation and regression tasks (Biau, Scornet, and Welbl 2019; Kocev, Vens et al. 2013;…”
Feature relevance quantification in explainable AI: A causality problem (2019, background): “Note, further, that our criticism concerns TreeExplainer in SHAP (Lundberg et al., 2018) more directly because it is really based on (3) rather than (4).”
On the Tractability of SHAP Explanations (2020, methodology): “Algorithm 1 Algorithm to compute value function v from (Lundberg, Erion, and Lee 2018) procedure EXPVALUE(x, S, tree = { v, a, b, t, r, d } ) procedure G(j) A Discussion on the TreeSHAP algorithm Lundberg, Erion, and Lee (2018) propose TreeSHAP, a variant of S HAP explanations for tree-based…”
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery (2018, positive): “The application of these values in Prescience uses fast estimation methods that we have developed to compute the Shapley values (that is, the estimated importance of features for a particular prediction) in real-time 20,21 .”
Problems with Shapley-value-based explanations as feature importance measures (2020, methodology): “TreeSHAP (Lundberg et al., 2018), an efficient algorithm for calculating SHAP values on additive tree-based models such as random forests and gradient boosting machines, can estimate EXS̄ |XS [f(xS ,XS̄)] by observing what proportion”
TabNet: Attentive Interpretable Tabular Learning (2019, methodology): “…(ii) they are highly interpretable in their basic form (e.g. by tracking decision nodes) and there are popular post-hoc explainability methods for their ensemble form, e.g. (Lundberg, Erion, and Lee 2018) – this is an important concern in many real-world applications; (iii) they are fast to train.”
Definitions, methods, and applications in interpretable machine learning (2019, methodology): “In the deep learning literature, a number of different approaches have been proposed to address this problem (70–79), with some methods for other models as well (80).”
Machine Unlearning for Random Forests (2020, methodology): “, 1984; Friedman, 2001) are popular and widely used machine learning models (Lundberg et al., 2018), mainly due to their predictive prowess on many classification and regression tasks (Kocev et al.”