Data Contamination: From Memorization to Exploitation

By Inbal Magar, Roy Schwartz. 2022

279 citations recorded by the source corpus.

Pretrained language models are typically trained on massive web-based datasets, which are often “contaminated” with downstream test sets. It is not clear to what extent models exploit the contaminated data for downstream tasks. We present a principled method to study this question. We pretrain BERT models on joint corpora of Wikipedia and labeled downstream datasets, and fine-tune them on the relevant task. Comparing performance between samples seen and unseen during pretraining enables us to define and quantify levels of memorization and exploitation.Experiments with two models and three downstream tasks show that exploitation exists in some cases, but in others the models memorize the contaminated data, but do not exploit it. We show that these two measures are affected by different factors such as the number of duplications of the contaminated data and the model size. Our results highlight the importance of analyzing massive web-scale datasets to verify that progress in NLP is obtained by better language understanding and not better data exploitation.

Subfields: Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Proving Test Set Contamination in Black Box Language Models (2023, methodology): “In our ablations below, we found that the tests are not particularly sensitive to these parameters, and we fix these parameters to avoid the possibility of p-hacking.”
Large Language Models as Analogical Reasoners (2023, background): “We focus on level-A problems published in 2023 to prevent test set contamination (Magar & Schwartz, 2022).”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “These issues may be more severe in recent LMs: the ever-growing pretraining datasets potentially expose the models to more evaluation instances, and the increasing sizes of recent LMs give them more ability to memorize these instances (Carlini et al., 2020; Magar and Schwartz, 2022).”
Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance (2022, background): “, 2022), classification tasks (Magar and Schwartz, 2022), and text gener-”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, result): “First, recent work (Dodge et al., 2021; Magar and Schwartz, 2022; OpenAI, 2023) suggests that examples from public datasets, like those used in our evaluation, may have leaked to the training data of recent LMs.”
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs (2024, background): “OpenAI’s GPT-3 (Brown et al., 2020; Magar and Schwartz, 2022), one of the first models with API-only access and limited training data disclosure.”
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023, methodology): “In this case, it has been shown that the high memorization abilities of LLMs can be used to generate portions of the training texts (Carlini et al., 2021; Magar and Schwartz, 2022).”
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, unknown): “Recent studies suggest that the remarkable performance of some LLMs might be rooted in their ability to memorize training data (Bender et al., 2021; Magar & Schwartz, 2022; Carlini et al., 2023; Biderman et al., 2023), rather than in generalization.”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, unknown): “8), the gold program is quite simple thus it is not surprising that et al., 2022; Peng et al., 2023; Kandpal et al., 2022a; Magar and Schwartz, 2022).”
Detecting Pretraining Data from Large Language Models (2023, background): “Additionally, Sainz et al. (2023); Magar & Schwartz (2022); Narayanan (2023) showed that the pretraining corpus may inadvertently include benchmark evaluation data, making it difficult to assess the effectiveness of these models.”