Problems with Shapley-value-based explanations as feature importance measures

By Indra Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Eduardo Scheidegger, Sorelle A. Friedler. 2020

484 citations recorded by the source corpus.

Game-theoretic formulations of feature importance have become popular as a way to "explain" machine learning models. These methods define a cooperative game between the features of a model and distribute influence among these input elements using some form of the game's unique Shapley values. Justification for these methods rests on two pillars: their desirable mathematical properties, and their applicability to specific motivations for explanations. We show that mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity, such as the need for causal reasoning. We also draw on additional literature to argue that Shapley values do not provide explanations which suit human-centric goals of explainability.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

On the Tractability of SHAP Explanations (2020, background): “More recently there has been a lot of work on the tradeoffs of variants of the original S HAP explanations, e.g., Sundararajan and Najmi (2020), Kumar et al. (2020), Janzing, Minorics, and Bloebaum (2020), Merrick and Taly (2020), and Aas, Jullum, and Løland (2019).”
Shapley values for feature selection: The good, the bad, and the axioms (2021, unknown): “There have, however, been a number of recent criticisms of the Shapley value in ML [37]–[39], suggesting to us that the Shapley value and its alternatives may be further developed considerably over the coming years.”
On the Privacy Risks of Model Explanations (2019, background): “[25] I E Kumar, S Venkatasubramanian, C Scheidegger, and S Friedler.”
Explaining by Removing: A Unified Framework for Model Explanation (2020, background): “Figure 13 shows more examples of MNIST explanations using the combination of the conditional distribution and Shapley value (i.e., LossSHAP).”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, background): “The change from unconditional to conditional permutation changes the respective interpretation method [78,7], or, in worst case, can break it [57,113,62].”
Algorithms to estimate Shapley value feature attributions (2022, background): “Having two popular types of Shapley value explanations has been cited as a weakness [22], but this issue is not unique to Shapley values; it is encountered by a large number of model explanation methods [26], and it is fundamental to understanding feature importance with correlated or statistically…”
Principles and Practice of Explainable Machine Learning (2020, background): “ible that the resulting scores are counterintuitive. On the other hand, some of these approaches, such as SHAP, come with some nice theoretical properties (although in practice they might be violated [50, 57]). ‹ Model simpliﬁcation comes with the immediate advantage and ﬂexibility of allowing to ap”
Epistemic values in feature importance methods: Lessons from feminist epistemology (2021, background): “The authors of SHAP recommend using interventional distributions if we want the explanation to be “true to the model”, and observational distributions if we want the explanation to be “true to the data” [14].”