How to Explain Individual Classification Decisions

By David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, K. Müller. 2009

1,191 citations recorded by the source corpus.

After building a classifier with modern tools of machine learning we typically have a black box at hand that is able to predict well for unseen data. Thus, we get an answer to the question what is the most likely label of a given unseen data point. However, most methods will provide no answer why the model predicted a particular label for a single instance and what features were most influential for that particular instance. The only method that is currently able to provide such explanations are decision trees. This paper proposes a procedure which (based on a set of assumptions) allows to explain the decisions of any classification method.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Intriguing properties of neural networks (2013, methodology): “decisions made by a model [1] and can be used to, for instance, identify the parts of the input which led to a correct classification of a given visual input instance (in other words, one can use a trained model for weakly-supervised localization).”
Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “To measure to what extent I→OR models exhibit this property, we use gradient-based attribution (Baehrens et al., 2010; Simonyan et al., 2014) to identify tokens important for label prediction, and the Remove and Retrain (ROAR) occlusion method (Hooker et al.”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “This has led to a substantial body of research that endeavors to make neural networks more “interpretable”, e.g. by attributing its prediction to a given part of its input (Baehrens et al., 2010; Sundararajan et al., 2017; ar X iv :2 00 4.”
Axiomatic Attribution for Deep Networks (2017, methodology): “Gradients (of the output with respect to the input) is a natural analog of the model coefficients for a deep network, and therefore the product of the gradient and feature values is a reasonable starting point for an attribution method (Baehrens et al., 2010; Simonyan et al., 2013); see the third co”
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps (2013, methodology): “In the case of a grey-scale image, the number of elements in w is equal to the number of pixels in I 0 , so the map can be computed as M ij = | w h ( i,j ) | , where h ( i, j ) is the index of the element of w , corresponding to the image pixel in the i -th row and j -th column.”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “They can be categorized as gradient-based (Springenberg et al., 2014; Sundararajan et al., 2017; Selvaraju et al., 2016; Baehrens et al., 2010; Rieger & Hansen, 2019), decomposition-based (Murdoch & Szlam, 2017; Shrikumar et al., 2016; Bach et al., 2015) and others (Dabkowski & Gal, 2017; Fong &…”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “Of the methods studied, gradients/sensitivity-based methods [127] and signal methods [71], [74] are input invariant while some attribution methods, such as integrated gradient [93], are not.”
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, methodology): “[5], use the derivative of a class score with respect to the image as an importance measure.”
Sanity Checks for Saliency Maps (2018, background): “The gradient explanation for an input x is Egrad(x) = ∂S ∂x [8, 22, 23].”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, neutral): “[87].”
Anchors: High-Precision Model-Agnostic Explanations (2018, methodology): “As a consequence of the need for users to understand the behavior of these models, interpretable machine learning has seen a resurgence in recent years, ranging from the design of novel globally-interpretable machine learning models (Lakkaraju, Bach, and Leskovec 2016; Ustun and Rudin 2015; Wang and”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Other methods that belong to this category are found in (Baehrens et al., 2010; Murdoch et al., 2018; Ribeiro et al., 2016b).”
Interpreting Deep Classifier by Visual Distillation of Dark Knowledge (2018, methodology): “An alternative is attribution methods that visualize how different parts of the input contribute to the ﬁnal output, such by generating sensitivity maps over the input (Bach et al., 2015; Baehrens et al., 2010; Selvaraju et al., 2016; Smilkov et al., 2017).”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, background): “1 OVERVIEW OF METHODS The gradient explanation for an input x is Egrad(x) = ∂S ∂x (Baehrens et al., 2010; Simonyan et al., 2013).”
Consistent Individualized Feature Attribution for Tree Ensembles (2018, methodology): “SHAP values are theoretically optimal, but like other model agnostic feature attribution methods [2, 9, 21, 27], they can be challenging to compute.”
A Survey of Methods for Explaining Black Box Models (2018, methodology): “In Reference [8] the authors propose a procedure based on Gaussian Process Classification (GDP), which allows explaining the decisions of any classification method through an explanation vector.”
Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (2017, methodology): “Not surprisingly, the development of techniques for “opening” black box models has recently received a lot of attention in the community [6, 35, 39, 5, 33, 25, 23, 30, 40, 11, 27].”