Influence Functions in Deep Learning Are Fragile

By Samyadeep Basu, Phillip E. Pope, Soheil Feizi. 2020

327 citations recorded by the source corpus.

Influence functions approximate the effect of training samples in test-time predictions and have a wide variety of applications in machine learning interpretability and uncertainty estimation. A commonly-used (first-order) influence function can be implemented efficiently as a post-hoc method requiring access only to the gradients and Hessian of the model. For linear models, influence functions are well-defined due to the convexity of the underlying loss function and are generally accurate even across difficult settings where model changes are fairly large such as estimating group influences. Influence functions, however, are not well-understood in the context of deep learning with non-convex loss functions. In this paper, we provide a comprehensive and large-scale empirical study of successes and failures of influence functions in neural network models trained on datasets such as Iris, MNIST, CIFAR-10 and ImageNet. Through our extensive experiments, we show that the network architecture, its depth and width, as well as the extent of model parameterization and regularization techniques have strong effects in the accuracy of influence functions. In particular, we find that (i) influence estimates are fairly accurate for shallow networks, while for deeper networks the estimates are often erroneous; (ii) for certain network architectures and datasets, training with weight-decay regularization is important to get high-quality influence estimates; and (iii) the accuracy of influence estimates can vary significantly depending on the examined test points. These results suggest that in general influence functions in deep learning are fragile and call for developing improved influence estimation methods to mitigate these issues in non-convex setups.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Foundation Models and Fair Use (2023, background): “…from difficulties in scaling due to high computational cost (e.g., leave-k-out retraining can be costly) (Feldman & Zhang, 2020; Zhang et al., 2021) or being inaccurate or erroneous when applied to complex but realistic model classes (Basu et al., 2020; Ghorbani et al., 2019; Søgaard et al., 2021).”
Counterfactual Memorization in Neural Language Models (2021, background): “However, the estimation was found to be computational expensive and fragile [Basu et al., 2021].”
Datamodels: Predicting Predictions from Training Data (2022, background): “On the other hand, Basu et al. (2021) finds that approximate influence functions scale poorly to deep neural network architectures; and Feldman & Zhang (2020) argue that understanding the dynamics of the penultimate layer is insufficient for understanding deep models’ decision mechanisms.”
LQF: Linear Quadratic Fine-Tuning (2020, unknown): “This analysis is similar to that of influence functions [25, 4], but in our case can be done without approximation since our loss is quadratic.”
Revisiting Methods for Finding Influential Examples (2021, methodology): “Note that the sensitivity of influence functions to damping factor has been studied previously (Basu, Pope, and Feizi 2021); but showing its (and other methods’) sensitivity to initialization, ordering, and batch size is, to the best of our knowledge, a novel contribution.”
On Memorization in Probabilistic Deep Generative Models (2021, background): “However, it has also been demonstrated that influence function estimates in deep learning models can be fragile [42].”
LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, background): “On the other hand, influence functions (Hampel, 1974) can counterfactually reason about model behaviors and training data (Koh & Liang, 2017), but some studies have found limitations (Bae et al., 2022) in its robustness (Basu et al., 2020) and effectiveness (Ilyas et al., 2022).”
TRAK: Attributing Model Behavior at Scale (2023, unknown): “, in the context of deep neural networks) (Basu et al., 2021; Ilyas et al., 2022; Bae et al., 2022).”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “cessful on classical machine learning algorithms, they do not effectively capture data dependence in modern deep neural networkswhich have highly non-convex loss surfaces [57].”
An Empirical Comparison of Instance Attribution Methods for NLP (2021, methodology): “Comparing explanation quality of gradient-based methods against simple similarity retrieval using leave-one-out (Basu et al., 2020) and randomized-test (Hanawa et al., 2021) analyses, we show that simpler methods are fairly competitive.”
Machine Unlearning of Features and Labels (2021, methodology): “Influence functions have then been used to trace bias in word embeddings back to documents [10, 19], determine reliable regions in learning models [46], and explain deep neural networks [7].”
Combining Feature and Instance Attribution to Detect Artifacts (2021, methodology): “Since TFA combines existing feature-and instance-based attribution methods, training-feature attribution inherits known issues with these techniques (Kindermans et al., 2019; Basu et al., 2020).”