The Explanation Game: Explaining Machine Learning Models with Cooperative Game Theory

By Luke Merrick, Ankur Taly. 2019

34 citations recorded by the source corpus.

A number of techniques have been proposed to explain a machine learning (ML) model's prediction by attributing it to the corresponding input features. Popular among these are techniques that apply the Shapley value method from cooperative game theory. While existing papers focus on the axiomatic motivation of Shapley values, and efficient techniques for computing them, they neither justify the game formulations used nor address the uncertainty implicit in their methods' outputs. For instance, the SHAP algorithm's formulation may give substantial attributions to features that play no role in a model. Furthermore, without infinite data and computation, SHAP attributions are approximations subject to hitherto uncharacterized uncertainty. In this work, we illustrate how subtle differences in the underlying game formulations of existing methods can cause large differences in attribution for a prediction. We then present a general game formulation that unifies existing methods. Using the primitive of single-reference games, we decompose the Shapley values of the general game formulation into Shapley values of single-reference games. This decomposition enables us to introduce confidence intervals to quantify the uncertainty in estimated attributions. Additionally, this decomposition enables contrastive explanations of a prediction through comparisons with different groups of reference inputs. We tie this idea to classic work on Norm Theory in cognitive psychology, and propose a general framework for generating explanations for ML models, called formulate, approximate, and explain (FAE).

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Principles and Practice of Explainable Machine Learning (2020, background): “On the other hand, some of these approaches, such as SHAP, come with some nice theoretical properties (although in practice they might be violated [50, 57]).”
Shapley values for feature selection: The good, the bad, and the axioms (2021, unknown): “Here, the meaning of worth depends strongly on the choice of evaluation function, and (less obviously) on the manner in which features are understood to be removed from the model (see [36] and [24, Section 4]).”
Explaining by Removing: A Unified Framework for Model Explanation (2020, background): “Lundberg and Lee (2017) view that features are marginalized out using their distribution from the training dataset (i.e., the marginal distribution), whereas Merrick and Taly (2019) view IME as marginalizing out features using a uniform distribution.”
Problems with Shapley-value-based explanations as feature importance measures (2020, methodology): “1), then the framework provided by Merrick and Taly [2019] provides a slightly different contrastive explanation: in their setting, the Shapley value assignment can be thought of as a set of answers to the question, “Why f(x) rather than f(r)?”, where r is chosen from the reference distribution.”