Interpretations are useful: penalizing explanations to align neural networks with prior knowledge

By Laura Rieger, Chandan Singh, W. James Murdoch, Bin Yu. 2019

258 citations recorded by the source corpus.

For an explanation of a deep learning model to be effective, it must provide both insight into a model and suggest a corresponding action in order to achieve some objective. Too often, the litany of proposed explainable deep learning methods stop at the first step, providing practitioners with insight into a model, but no way to act on it. In this paper, we propose contextual decomposition explanation penalization (CDEP), a method which enables practitioners to leverage existing explanation methods in order to increase the predictive accuracy of deep learning models. In particular, when shown that a model has incorrectly assigned importance to some features, CDEP enables practitioners to correct these errors by directly regularizing the provided explanations. Using explanations provided by contextual decomposition (CD) (Murdoch et al., 2018), we demonstrate the ability of our method to increase performance on an array of toy and real datasets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Refining Language Models with Compositional Explanations (2021, background): “Despite these initial successes, existing model regularization methods have limited capacity in conveying complex human feedback regarding spurious patterns, and limited regularization strength as the regularization term is enforced on only instances associated with human feedback (while the vast am”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “…we applied a simple mean-squared error between the predicted saliency maps and human visual explanations, similar to the explanation loss terms in [34, 39]: where e i is the predicted saliency mask for state s i and A i is the preprocessed (resized and stacked) human explanation binary mask, in…”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, unknown): “line with human explanations [108], [109].”
Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence (2020, background): “Contextual Decomposition Explanation Penalization (CDEP) [150] adds a term the optimization objective that imposes a constraint on the parameters of a neural network so they learn how to produce good explanations in addition to predicting the correct value.”
Explanation-Based Human Debugging of NLP Models: A Survey (2021, unknown): “This has been experimented with in the computer vision domain (Rieger et al., 2020), and its use in the EBHD setting in NLP could be an interesting direction to explore.”
Debugging Tests for Model Explanations (2020, background): “Correcting bugs can also be achieved by penalizing feature attributions during training (Erion et al., 2019; Rieger et al., 2020; Ross et al., 2017) or clustering (Lapuschkin et al.”
Explainable Machine Learning for Scientific Insights and Discoveries (2019, background): “Reference [79] proposes contextual decomposition explanation penalization, which constrains a classification or regression result to more accurate and more scientifically plausible results by leveraging the explained outcome of interpretation tools.”
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems (2020, methodology): “We report AUROC for ISIC, as is standard [41].”
Improving performance of deep learning models with axiomatic attribution priors and expected gradients (2020, background): “More recent work discusses the need for priors that incorporate human intuition in order to develop robust and interpretable models [13].”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, background): “…the explanations, which is typically done by interpreting a prediction post-hoc through explanation techniques and optimize the distance between the obtained explanation and ground truth explanation (Liu and Avci, 2019; Rieger et al., 2020; Plumb et al., 2020; Erion et al., 2021; Yao et al., 2021).”
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (2022, background): “Another line of work seeks to align the reasoning process of a trained model with the explanations, which is typically done by interpreting a prediction post-hoc through explanation techniques and optimizing the distance between the obtained explanation and ground truth explanation (Liu and Avci, 20”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, methodology): “[42] propose an explanation penalization loss similar to [43, 47, 44], focusing on Contextual Decomposition [35] as explanation method.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, methodology): “For example, explainable AI is used to analyze whether model f is fair [78] or biased [173, 246], whether f is right for the right reasons [221], or whether f is overfitting [41].”
Transformation Importance with Applications to Cosmology (2020, methodology): “In the work here, the choice of attribution method attr is CD, as it can disentangle the importance of features and their interactions, and has been rigorously evaluated using real data [1], human experiments [23], and during model training [24].”