Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions

By Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, Yoav Goldberg. 2022

67 citations recorded by the source corpus.

Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. But what exactly in the training data causes a model to make a certain prediction? We seek to answer this question by providing a language for describing how training data influences predictions, through a causal framework. Importantly, our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone. Addressing the problem of extracting factual knowledge from pretrained language models (PLMs), we focus on simple data statistics such as co-occurrence counts and show that these statistics do influence the predictions of PLMs, suggesting that such models rely on shallow heuristics. Our causal framework and our results demonstrate the importance of studying datasets and the benefits of causality for understanding NLP models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Recent work has identified the frequency of specific facts within a corpus as an important factor in how likely a model is capable of applying that fact in response to a natural language question (Razeghi et al., 2022; Elazar et al., 2022; Kandpal et al., 2022; Mallen et al., 2022).”
Counterfactual Memorization in Neural Language Models (2021, methodology): “Alternative methods also looked at simple data statistics (e.g. co-occurrence counts) without model re-training to infer the causal effects on language models’ predictions [Elazar et al., 2022].”
Large Language Models Struggle to Learn Long-Tail Knowledge (2022, background): “The most similar work to ours in this context is Elazar et al. (2022), who use causal inference to measure the effect of pre-training data statistics on QA performance.”
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2021, unknown): “Other papers in this direction include Akyurek et al. (2022), Han and Tsvetkov (2022), and Elazar et al. (2022).”