Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations

By Andrew Slavin Ross, Michael C. Hughes, Finale Doshi-Velez. 2017

691 citations recorded by the source corpus.

Neural networks are among the most accurate supervised learning methods in use today, but their opacity makes them difficult to trust in critical applications, especially when conditions in training differ from those in test. Recent work on explanations for black-box models has produced tools (e.g. LIME) to show the implicit rules behind predictions, which can help us identify when models are right for the wrong reasons. However, these methods do not scale to explaining entire datasets and cannot correct the problems they reveal. We introduce a method for efficiently explaining and regularizing differentiable models by examining and selectively penalizing their input gradients, which provide a normal to the decision boundary. We apply these penalties both based on expert annotation and in an unsupervised fashion that encourages diverse models with qualitatively different decision boundaries for the same classification problem. On multiple datasets, we show our approach generates faithful explanations and models that generalize much better when conditions differ between training and test.

Subfields: Artificial Intelligence, Model Architectures, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Safety, Robustness & Interpretability

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “, 2019), and methods that search for validationequivalent predictors that are “right for the right reasons” (Ross et al., 2017).”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “, 2017; 2018), or through placing priors over how features are weighted or extracted (Zaidan et al., 2007; Small et al., 2011; Zhang et al., 2016; Ross et al., 2017; Bao et al., 2018; Selvaraju et al., 2019; Liang et al., 2020; Stammer et al., 2020).”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “Especially relevant is work on supervising models using explanations (Ross et al., 2017), which, similar to our annotations, identify parts of the input that are important for prediction (Lei et al., 2016; Ribeiro et al., 2016).”
Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “However, [23] consider somewhat simple data and generating explanations for more complex tasks (like image captioning) is a difficult and active area of research [24,25,26,27,28,29].”
Attention is not Explanation (2019, methodology): “These include approaches that measure feature importance based on gradient information (Ross et al., 2017; Sundararajan et al., 2017) (aligned with the gradient-based measures that we have used here), and methods based on representation erasure (Li et al.”
Attention is not not Explanation (2019, background): “Citing Ross et al. (2017), Jain and Wallace’s requisite for attention distributions to be used as explanation is that there must only exist one or a few closely-related correct explanations for a model prediction.”
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “This is the technique proposed by Ross [60], where input sensitivity of a network is adjusted and measured in order to create networks that are “right for the right reasons;” the method can be used to steer the internal reasoning learned by a network.”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, methodology): “A similar idea to Eq 1 has been proposed in previous/concurrent work, where the choice of explanation method uses a gradient-based attribution method (Ross et al., 2017; Erion et al., 2019).”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “While there are several different techniques for obtaining local-explanations [96]–[100], recent work by Lundberg and Lee [101] have shown that there are equivalences among these techniques.”
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, background): “Ross, Hughes, and Doshi-Velez (2017) penalize input sensitivity to features marked as less relevant.”
Interpretable Convolutional Neural Networks (2017, background): “[19] required people to label dimensions of the input that were related to each output, in order to learn a better model.”
Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning (2021, background): “For example, there may exist inconsistency (Ross et al., 2017) in the explanations since 1University of Iowa 2University of Texas at Austin.”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, background): “Further research could investigate to what extent undesired prototypes can be ‘fixed’ with a human-in-the-loop that replaces them with a manually selected patch, in order to create a model that is completely “right for the right reasons” [42].”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “In fact, the use of human visual explanation has been investigated by recent works in several supervised learning tasks [39, 35, 34].”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, methodology): “Instead of regularizing the gradients with respect to X, as originally described in [4], we regularize the gradients of the final convolutional layer h, corresponding to Gradient weighted Class Activation Maps (grad-Cam) ([20], cf.”
Explainable machine learning in deployment (2019, background): “[51] attempt to develop a technique for dynamic explanations, wherein the human can guide the model towards learning the correct explanation.”
Learning Credible Models (2017, background): “[25] constrains the input gradient of features that are believed not to be relevant in a neural network.”