An unexpected unity among methods for interpreting model predictions

By Scott M. Lundberg, Su-In Lee. 2016

120 citations recorded by the source corpus.

Understanding why a model made a certain prediction is crucial in many data science fields. Interpretable predictions engender appropriate trust and provide insight into how the model may be improved. However, with large modern datasets the best accuracy is often achieved by complex models even experts struggle to interpret, which creates a tension between accuracy and interpretability. Recently, several methods have been proposed for interpreting predictions from complex models by estimating the importance of input features. Here, we present how a model-agnostic additive representation of the importance of input features unifies current methods. This representation is optimal, in the sense that it is the only set of additive values that satisfies important properties. We show how we can leverage these properties to create novel visual explanations of model predictions. The thread of unity that this representation weaves through the literature indicates that there are common principles to be learned about the interpretation of model predictions that apply in many scenarios.

Subfields: Artificial Intelligence, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical Learning Theory

Citations in the corpus

Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, background): “Other post-hoc interpretations typically typically evaluate the sensitivity of predictions to local perturbations of inputs or the input gradient (Ribeiro, Singh, and Guestrin 2016; Sel-varaju et al. 2016; Adler et al. 2016; Lundberg and Lee 2016; Erhan et al. 2009).”
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, background): “However, within the machine learning community recently there has been convergence [Lundberg and Lee, 2016] around local counterfactual explanations, where we show how perturbing an input x in various ways will affect the model’s prediction ŷ.”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, background): “Local explainers instead focus on the arguably more approachable task of explaining individual predictions [9].”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “Lundberg and Lee [342] noted that several importance scoring methods including integrated gradients and LIME could all be considered approximations to Shapely values [343], which have a long history in game theory for assigning contributions to players in cooperative games.”
Interpreting Blackbox Models via Model Extraction (2017, methodology): “Similarly, [32] uses the Shapley value to determine the most inﬂuential features for a given prediction.”
Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs (2018, methodology): “It can also be viewed as an approximation to Shapely values, as discussed in Lundberg & Lee (2016) and Shrikumar et al. (2017).”
Maximum entropy methods for extracting the learned features of deep neural networks (2017, unknown): “This approach to assigning importance has been motived as an approximation to Shapley values, which describe distribution of credit in game theory [9].”
Explainable machine learning predictions to help anesthesiologists prevent hypoxemia during surgery (2017, methodology): “…how to retain interpretability, even when complex models such as nonparametric methods or deep learning are used, by developing a method to provide theoretically justified explanations of model predictions building on recent advances in model-agnostic prediction explanation methods (14–16) .”
Explanatory Interactive Machine Learning (2019, methodology): “Local approaches instead interpret individual predictions [19].”
MAGIX: Model Agnostic Globally Interpretable Explanations (2017, methodology): “[LL16] extend a prediction explanation method based on the Shapley value from game theory.”
Stakeholders in Explainable AI (2018, methodology): “They may use open source libraries created for generating explanations; some well-known and widely-used examples include LIME (Ribeiro, Singh, and Guestrin 2016), deep Taylor decomposition (Montavon et al. 2016), inﬂuence functions (Koh and Liang 2017) and Shapley Additive Explanations (Lundberg and”
Predicting mortgage default using convolutional neural networks (2018, methodology): “…of transparency in many applications is considered a major drawback, the development of methods for explaining and interpreting deep learning models has recently attracted increased attention, see e.g. Shrikumar et al. (2016), Samek et al. (2017), Lundberg and Lee (2016), and Ribeiro et al. (2016).”
Adversarial Examples on Graph Data: Deep Insights into Attack and Defense (2019, background): “Integrated gradients approximate Shapley values [Hart, 1989; Lundberg and Lee, 2016] by integrating partial gradients with respect to input features from reference input to the actual input.”