The many Shapley values for model explanation
By Mukund Sundararajan, Amir-Homayoon Najmi. 2019
847 citations recorded by the source corpus.
The Shapley value has become a popular method to attribute the prediction of a machine-learning model on an input to its base features. The use of the Shapley value is justified by citing [16] showing that it is the \emph{unique} method that satisfies certain good properties (\emph{axioms}). There are, however, a multiplicity of ways in which the Shapley value is operationalized in the attribution problem. These differ in how they reference the model, the training data, and the explanation context. These give very different results, rendering the uniqueness result meaningless. Furthermore, we find that previously proposed approaches can produce counterintuitive attributions in theory and in practice---for instance, they can assign non-zero attributions to features that are not even referenced by the model. In this paper, we use the axiomatic approach to study the differences between some of the many operationalizations of the Shapley value for attribution, and propose a technique called Baseline Shapley (BShap) that is backed by a proper uniqueness result. We also contrast BShap with Integrated Gradients, another extension of Shapley value to the continuous setting.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Su-In Lee
- Percy Liang
- Mohit Bansal
- Scott M. Lundberg
- Yoshua Bengio
- Luke S. Zettlemoyer
- Jianfeng Gao
- Noah A. Smith
In Safety, Robustness & Interpretability
- Ankur Taly
- Christoph Molnar
- Bernd Bischl
- Su-In Lee
- Scott M. Lundberg
- Giuseppe Casalicchio
- Luke Merrick
- Percy Liang
In Game & Decision Theory
- Lenon Minorics
- Ian Covert
- Percy Liang
- Bernd Bischl
- Dominik Janzing
- Tatsunori B. Hashimoto
- Su-In Lee
- Scott M. Lundberg
Citations in the corpus
- Foundation Models and Fair Use (2023, methodology): “38As Feldman & Zhang (2020, at 5-6) note, instance attribution mechanisms could be related to Shapley values—though Shapley values are typically used for attribution of model outputs to input features, e.g., Sundararajan & Najmi (2020).”
- Explaining by Removing: A Unified Framework for Model Explanation (2020, methodology): “(2) Sometimes referred to as the baseline method (Sundararajan and Najmi, 2019), this is a generalization of the previous approach, and in some cases features may be given different default values (e.g., their mean).”
- On the Privacy Risks of Model Explanations (2019, background): “Perturbation-based explanation methods have been criticized for not following the distribution of the training data and violating the manifold hypothesis [28, 52].”
- On the Tractability of SHAP Explanations (2020, background): “More recently there has been a lot of work on the tradeoffs of variants of the original S HAP explanations, e.g., Sundararajan and Najmi (2020), Kumar et al. (2020), Janzing, Minorics, and Bloebaum (2020), Merrick and Taly (2020), and Aas, Jullum, and Løland (2019).”
- The Explanation Game: Explaining Machine Learning Models with Cooperative Game Theory (2019, background): “We refer to [21] for a comprehensive analysis of the axiomatic guarantees of various game formulations.”
- The explanation game: a formal framework for interpretable machine learning (2020, neutral): “See Sundararajan and Najmi (2019) for an overview.”
- Feature relevance quantification in explainable AI: A causality problem (2019, methodology): “Contrary to LIME, DeepLIFT and LRP, this attribution method has the advantage that all of the following 5 properties are satisfied (see Sundararajan et al. (2017) and Aas et al.”
- Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, methodology): “We have a better understanding of weaknesses of IML methods in general [75,79], but also specifically for methods such as permutation feature importance [51,110,7,111], Shapley values [57,113], counterfactual explanations [63], partial dependence plots [51,50,7] and saliency maps [2].”
- From Local Explanations to Global Understanding with Explainable AI for Trees (2020, methodology): “An equivalent formulation is the randomized baseline method discussed in [10].”