Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions

By Xiaochuang Han, Byron C. Wallace, Yulia Tsvetkov. 2020

210 citations recorded by the source corpus.

Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting important words in the corresponding input text. While this might be useful for tasks where decisions are explicitly influenced by individual tokens in the input, we suspect that such highlighting is not suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers. Influence functions explain the decisions of a model by identifying influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which ‘saliency maps’ may not have clear interpretation. Furthermore, we develop a new quantitative measure based on influence functions that can reveal artifacts in training data.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “, 2020) use neural models for textual entailment whose reasoning is opaque to humans despite advances in interpretablity (Han et al., 2020).”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “…as we can now understand how important concepts from the training set inﬂuenced the model decision to predict the label of a new input, providing more granularity than methods that use entire samples from the training data for post-hoc interpretability (Koh and Liang, 2017; Han et al., 2020).”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, background): “Also related are interpretability methods that explain predictions (Han et al., 2020; Wiegreffe and Pinter, 2019), however, our focus is entirely different: we do not interpret individual predictions, but aim to understand the mechanism of transformers.”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “Han et al.(78) demonstrate the application of influence functions, originally introduced by Koh and Liang(79) as a way to identify the influence of particular training examples onmodel predictions, to the discovery of data artifacts.”
Explaining and Improving Model Behavior with k Nearest Neighbor Representations (2020, methodology): “Our method is closely related to the work on inﬂuence functions (Koh and Liang 2017) which has been recently extended by Han, Wallace, and Tsvetkov (2020) to neural text classiﬁers.”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “Han et al. [47] perform a similar experiment, but instead removes observations based on importance and then measures the performance difference.”
Explanation-Based Human Debugging of NLP Models: A Survey (2021, background): “, 2015), and to find potential causes of errors (Han et al., 2020).”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “Prototypes (Parts of) Representative examples, including concepts [125], influential training instances [90], prototypical parts [35, 174], nearest neighbors and criticisms [124].”
DataComp: In search of the next generation of multimodal datasets (2023, methodology): “[53] Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov.”
A Review on Language Models as Knowledge Bases (2022, background): “Han et al. (2020) applies influence functions to LMs, finding that influence functions may be better suited for more complex tasks, such as natural language inference, despite these methods being computationally prohibitive (Pezeshkpour et al., 2021b).”
An Empirical Comparison of Instance Attribution Methods for NLP (2021, methodology): “Instance attribution methods constitute a promising approach to better understanding how modern NLP models come to make the predictions that they do (Han et al., 2020; Koh and Liang, 2017).”
Combining Feature and Instance Attribution to Detect Artifacts (2021, methodology): “To the best of our knowledge, only one previous work (Han et al., 2020) set out to provide a methodical approach to artifact detection.”