Understanding Black-box Predictions via Influence Functions
By Pang Wei Koh, Percy Liang. 2017
3,706 citations recorded by the source corpus.
How can we explain the predictions of a black-box model? In this paper, we use influence functions — a classic technique from robust statistics — to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks.
Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Quoc V. Le
- Noah A. Smith
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Luke S. Zettlemoyer
- Percy Liang
- Noah A. Smith
- Hannaneh Hajishirzi
- Graham Neubig
- Jianfeng Gao
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Nicolas Papernot
- Eric Wallace
- Bo Li
- Florian Tramèr
- Percy Liang
- Matthias Hein
- Longo Luca
Citations in the corpus
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “, 2019) or that toxic examples may be more salient for the model and hence harder to unlearn (Koh and Liang, 2017).”
- Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Influence functions (Koh & Liang, 2017), forgetting events (Toneva et al., 2018), and the Data Shapley (Ghorbani & Zou, 2019; Jia et al., 2019) can all be used to assign point-wise estimates of importance to data instances based on their contribution to the decision boundary.”
- SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “is advantageous as we can now understand how important concepts from the training set influenced the model decision to predict the label of a new input, providing more granularity than methods that use entire samples from the training data for posthoc interpretability (Koh and Liang, 2017; Han et al”
- Foundation Models and Fair Use (2023, methodology): “…attribution refers to methods that assign attribution scores to training examples to understand the contribution of individual examples (or group of examples) to (test-time) model predictions (Koh & Liang, 2017; Ghorbani & Zou, 2019; Jia et al., 2019; Pezeshkpour et al., 2021; Ilyas et al., 2022).”
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “each class represented by clusters of image patches and [30] which identifies training examples most responsible for a prediction.”
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, background): “Recent work addresses this requirement either through saliency-map visualizations [61, 77, 49], attention mechanism [75, 79, 51, 14], or other analysis [30, 39, 56, 58].”
- Semantically Equivalent Adversarial Rules for Debugging NLP models (2018, neutral): “, 2011), or find labeling errors (Zhang et al., 2018; Koh and Liang, 2017).”
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “Recently, preliminary attempts towards poisoning deep networks have also been re-ported, showing the first adversarial training examples against deep learners [30, 31].”
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “In [221] Representative works in this area can be found in [222] , which present a portfolio of visualization techniques to help in the explanation of a black-box ML model built upon the set of extended techniques mentioned earlier (Global SA).”
- Time-series forecasting with deep learning: a survey (2020, methodology): “Next, gradient-based methods—such as saliency maps [71,72] and influence functions [73]—have been proposed, which analyse network gradients to determine which input features have the greatest impact on loss functions.”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Across most FL settings, it is assumed that there is no single, central authority that owns or verifies the training data or user hardware, and it has been argued by many recent studies that FL lends itself to new adversarial attacks during decentralized model training [12, 13, 14, 15, 16, 17, 18, 1”
- Backdoor Attacks on Federated Meta-Learning (2020, background): “Given the requirement that users of federated learning share only updates of the model instead of training data [1], techniques certifying that training examples are correctly classified [34, 35] are not applicable in this context, and thus defense remains a challenging problem.”
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “In contrast, poisoning attacks target the training process by maliciously modifying part of training data to cause the trained model to misbehave on some test inputs [6, 27, 43].”
- Analyzing Federated Learning through an Adversarial Lens (2018, background): “We focus on an adversary which directly performs model poisoning instead of data poisoning [3, 21, 17, 27, 17, 13, 7, 11] as the agents’ data is never shared with the server.”
- Dataset Distillation (2018, methodology): “Researchers have presented various approaches for understanding and visualizing learned models (Zeiler & Fergus, 2014; Zhou et al., 2015; Mahendran & Vedaldi, 2015; Bau et al., 2017; Koh & Liang, 2017).”