A Unified Approach to Interpreting Model Predictions
By Scott M. Lundberg, Su-In Lee. 2017
37,902 citations recorded by the source corpus.
Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Luke S. Zettlemoyer
- Noah A. Smith
- Jianfeng Gao
- Quoc V. Le
- Yejin Choi
In Safety, Robustness & Interpretability
- Longo Luca
- Giulia Vilone
- Been Kim
- Wojciech Samek
- K. Müller
- Cynthia Rudin
- Grgoire Montavon
- Lars Kai Hansen
Citations in the corpus
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “One fruitful contribution to this path is that of [224] called SHAP (SHapley Additive exPlanations).”
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “Despite intense interest and collective group efforts, we are only seeing limited progress in definitions [8], frameworks [9], visualization [10], and limited experimentation [11].”
- STRIP: a defence against trojan attacks on deep neural networks (2019, methodology): “For examples, given an input, the model shall identify which features contribute the most to the model outputs [5], [22], [23].”
- Explaining Models by Propagating Shapley Values of Local Components (2019, methodology): “Popular model agnostic explanation methods that also aim to obtain SHAP values are KernelSHAP (Lundberg and Lee 2017) and IME (ˇStrumbelj and Kononenko 2014).”
- A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “A linear combination model with xi ∈ {0, 1} has been referred to as the additive feature attribution method [39].”
- Fairwashing: the risk of rationalization (2019, methodology): “Approaches such as LIME [30] and SHAP [26] belong to this class.”
- Sanity Checks for Saliency Maps (2018, methodology): “Second, we consider methods that approximate an element-wise product of input and gradient, as several local explanations do [33, 18].”
- On the Art and Science of Machine Learning Explanations (2018, background): “Shapley explanations, including tree shap and even certain implementations of LIME, are a class of additive, consistent local feature contribution measures with long-standing theoretical support [20].”
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “While there are several different techniques for obtaining local-explanations [96]–[100], recent work by Lundberg and Lee [101] have shown that there are equivalences among these techniques.”
- Visualizing the Feature Importance for Black Box Models (2018, background): “Further developments, visualizations, and generalizations were introduced by [15, 16].”
- Explanations of model predictions with live and breakDown packages (2018, methodology): “• When parameters (kernel and regularization term) are chosen as in Lundberg and Lee (2017), lime produces estimates of Shapley values , while other choices of kernel and penalty term lead to inconsistent results.”
- Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, background): “…obtaining local-explanations (Ribeiro et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Pieter-Jan Kindermans, 2018), recent work by Ancona et al. (2017) and Lundberg & Lee (2017) both show that there are equivalences among several of the previously proposed methods.”
- Consistent Individualized Feature Attribution for Tree Ensembles (2018, neutral): “additive feature attribution methods [16].”
- Interpreting CNNs via Decision Trees (2018, methodology): “As discussed in [22, 24], we can use a piecewise linear representation to represent the function of cascaded FC layers and ReLU layers, as follows.”