Studying Large Language Model Generalization with Influence Functions

By Roger Baker Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez and 7 more. arXiv.org, 2023

343 citations recorded by the source corpus.

When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior? Influence functions aim to answer a counterfactual: how would the model's parameters (and hence its outputs) change if a given sequence were added to the training set? While influence functions have produced insights for small models, they are difficult to scale to large language models (LLMs) due to the difficulty of computing an inverse-Hessian-vector product (IHVP). We use the Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) approximation to scale influence functions up to LLMs with up to 52 billion parameters. In our experiments, EK-FAC achieves similar accuracy to traditional influence function estimators despite the IHVP computation being orders of magnitude faster. We investigate two algorithmic techniques to reduce the cost of computing gradients of candidate training sequences: TF-IDF filtering and query batching. We use influence functions to investigate the generalization patterns of LLMs, including the sparsity of the influence patterns, increasing abstraction with scale, math and programming abilities, cross-lingual generalization, and role-playing behavior. Despite many apparently sophisticated forms of generalization, we identify a surprising limitation: influences decay to near-zero when the order of key phrases is flipped. Overall, influence functions give us a powerful new tool for studying the generalization properties of LLMs.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/04a96b66705858c988edfcb73191c1da7d54abfb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fan Yang
2. Ninghao Liu
3. Mengnan Du
4. Meg Tong
5. Huiqi Deng
6. Yihua Zhang
7. Rongwu Xu
8. Jiancheng Liu

Citations in the corpus

• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (2023, background): “Grosse et al. (2023) provide complementary evidence for the Reversal Curse.”
• SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation (2023, methodology): “The application of data attribution includes model explanation (Jeyakumar et al., 2020; Grosse et al., 2023), debugging (Ilyas et al., 2022), efficient training (Xie et al., 2023), and improving model generalization (Jain et al., 2023).”
• Knowledge Conflicts for LLMs: A Survey (2024, positive): “…et al., 2022; Singhal et al., 2022), the vast and often unreliable nature of internet-sourced training data (Bender et al., 2021; Weidinger et al., 2021) can lead to the memorization and amplification of inaccuracies (Lin et al., 2022; Elazar et al., 2022; Lam et al., 2022; Grosse et al., 2023).”
• Rethinking machine unlearning for large language models (2024, neutral): “However, evidence shows that existing state-of-the-art MIA methods for LLMs are limited in their ability to effectively distinguish membership and non-membership (Duan et al., 2024), suggesting opportunities for further research.”
• Explainability for Large Language Models: A Survey (2023, background): “The result indicates that as model scale increases, influential sequences are better at capturing the reasoning process for queries, whereas smaller models often provide semantically unrelated pieces of information (Grosse et al., 2023).”
• Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction (2025, unknown): “For LLM safety, applications extend beyond jailbreaks to influence functions and steering vectors (Grosse et al., 2023; Subramani et al., 2022), while advancing our understanding of human preferences (Li et al., 2024).”