Impact of Pretraining Term Frequencies on Few-Shot Reasoning

By Yasaman Razeghi, Robert L Logan IV, Matt Gardner, Sameer Singh. 2022

185 citations recorded by the source corpus.

Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. However, the extent to which this extrapolation relies on robust reasoning is unclear. In this paper, we investigate how well these models reason with terms that are less frequent in the pretraining data. In particular, we examine the correlations between the model performance on test instances and the frequency of terms from those instances in the pretraining data. We measure the strength of this correlation for a number of GPT-based language models (pretrained on the Pile dataset) on various numerical deduction tasks (e.g., arithmetic and unit conversion). Our results consistently demonstrate that models are more accurate on instances whose terms are more prevalent, in some cases above $70\%$ (absolute) more accurate on the top 10\% frequent terms in comparison to the bottom 10\%. Overall, although LMs exhibit strong performance at few-shot numerical reasoning tasks, our results raise the question of how much models actually generalize beyond pretraining data, and we encourage researchers to take the pretraining data into account when interpreting evaluation results.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

Emergent Abilities of Large Language Models (2022, positive): “Training long enough on a large-enough dataset has been shown to be key for the ability of language models to acquire syntactic, semantic, and other world knowledge (Zhang et al., 2021; Wei et al., 2021b; Razeghi et al., 2022).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Razeghi et al. (2022) show that there is a strong correlation between the frequency of a numerical equation in the Pile and GPT-J’s performance on that equation, and we see no reason this would not hold in GPT-NeoX 20B, FairSeq, and GPT-3.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “Emergence of the Impact of Pretraining Frequencies Recent work has identified the frequency of specific facts within a corpus as an important factor in how likely a model is capable of applying that fact in response to a natural language question (Razeghi et al., 2022; Elazar et al., 2022; Kandpal e”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “Razeghi et al. (2022) show that in-context learning performance is highly correlated with term frequencies in the pretraining data.”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Razeghi et al. [57] show that in-context learning for numerical reasoning tasks is better for instances whose terms are more prevalent in training data.”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, background): “However, this behavior, often backed up by corpus counts (Razeghi et al., 2022), only reflects a correlation Figure 1: In this figure, we showcase one of our hypotheses of how the model is influenced by the co-occurrences between subjects and objects”
Demystifying Prompts in Language Models via Perplexity Estimation (2022, background): “Razeghi et al. (2022) study numeric deduction tasks, and examine the correlations be-tween the model performance on specific test instances and the frequency of terms from those instances in the pretraining data.”
Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022, background): “…large language models may not actually be performing genuine in-context learning, and simply draw on examples seen in training (Min et al., 2022; Razeghi et al., 2022; Xie et al., 2021) – our experiments show that naturalistic distributional properties can give rise to a capacity for in-context…”
Language models show human-like content effects on reasoning (2022, background): “, 2021), or reason more accurately about frequently-occurring numbers (Razeghi et al., 2022), have been cited to “rais[e] questions on the extent to which these models are actually reasoning” (ibid, emphasis ours).”
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, methodology): “…popularity of the entities measured by Wikipedia monthly page views as a proxy for how frequently the entities are likely to be discussed on the web, instead of using the occurrence of entities or strings in the pretraining corpus (Carlini et al., 2022; Kandpal et al., 2022; Razeghi et al., 2022).”
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024, unknown): “Razeghi et al. (2022)”
Data Contamination: From Memorization to Exploitation (2022, background): “These works study the impact of duplication of training sequence on regeneration of the sequence (Carlini et al., 2022; Kandpal et al., 2022), and the effect on few-shot numerical reasoning (Razeghi et al., 2022).”
Large Language Models Are Reasoning Teachers (2022, background): “Additionally, previous work shows that GPT-3 exhibits a performance gap between instances including terms that are frequent in the pretraining corpus, and instances including less frequent terms (Razeghi et al., 2022).”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “Razeghi et al. (2022) conﬁrm this memorization trend for simple arithmetic problems.”
Large Language Models Struggle to Learn Long-Tail Knowledge (2022, background): “For example, Razeghi et al. (2022) show mathematical reasoning capabilities can be correlated with training data frequency, and Shin et al. (2022) and Han & Tsvetkov (2022) show that training corpus source can inﬂuence few-shot accuracies.”
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2022, background): “, Razeghi et al. (2022) investigates the correlation between the model performance and the frequency of terms from the test instances in the pretraining data.”