Feature relevance quantification in explainable AI: A causality problem

By Dominik Janzing, Lenon Minorics, Patrick Blöbaum. 2019

367 citations recorded by the source corpus.

We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between observational and interventional conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that unconditional rather than conditional expectations provide the right notion of dropping features in contradiction to the theoretical justification of the software package SHAP. Parts of SHAP are unaffected because unconditional expectations (which we argue to be conceptually right) are used as approximation for the conditional ones, which encouraged others to `improve' SHAP in a way that we believe to be flawed.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Explaining by Removing: A Unified Framework for Model Explanation (2020, methodology): “When users are more interested in the model’s mechanism for calculating predictions, other removal approaches may be preferable, such as interventional approaches motivated by a causal analysis of the model (Janzing et al., 2019; Heskes et al., 2020).”
From Local Explanations to Global Understanding with Explainable AI for Trees (2020, methodology): “To compute the impact of a specific feature subset during the Shapley value calculation, TreeExplainer uses interventional expectations over a user-sup-plied background dataset 11 .”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, methodology): “We have a better understanding of weaknesses of IML methods in general [75,79], but also specifically for methods such as permutation feature importance [51,110,7,111], Shapley values [57,113], counterfactual explanations [63], partial dependence plots [51,50,7] and saliency maps [2].”
On the Tractability of SHAP Explanations (2020, methodology): “More recently there has been a lot of work on the tradeoffs of variants of the original S HAP explanations, e.g., Sundararajan and Najmi (2020), Kumar et al. (2020), Janzing, Minorics, and Bloebaum (2020), Merrick and Taly (2020), and Aas, Jullum, and Løland (2019).”