Deduplicating Training Data Makes Language Models Better

By Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini. 2021

926 citations recorded by the source corpus.

We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings.As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data.We develop two tools that allow us to deduplicate training datasets—for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times.Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer training steps to achieve the same or better accuracy.We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation.Code for deduplication is released at https://github.com/google-research/deduplicate-text-datasets.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Additionally, due to the presence of near-duplicate text in web-derived corpora, some passages (with minor variations) are seen multiple times during training (Lee et al., 2021).”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We identify such 10(+)-skipgrams with suffix arrays implemented using a variation of the library from Lee et al. (2022), modified to work on a PySpark cluster (effectively without random access to disk).”
Improving language models by retrieving from trillions of tokens (2021, background): “6), addressing the problem of test set leakage (Lee et al., 2021).”
Scaling Up Models and Data with t5x and seqio (2022, background): “This avoids repeating data on intentional or unintentional (e.g., due to preemption) training restarts, which can lead to reduced performance and memorization (Lee et al., 2022).”
Quantifying Memorization Across Neural Language Models (2022, methodology): “Because the frequency of training data duplication follows an exponential distribution (Lee et al., 2021), a fully random sample of only 50,000 sequences (accounting for ≤ 0.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “However, we did not find a single paper that estimated how many annotations should be collected, and most opted for the “typical” 100 data points which, as pointed out above, may be insufficient (van der Lee et al., 2021).”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, methodology): “Fortunately, this defense comes at little-to-no cost in model performance, as training on deduplicated data does not harm language modeling perplexity (Lee et al., 2021).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “Lee et al. (2021) releases the deduplication code that they used, which we intend to use to explore this question in more detail in the future.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “…FairSeq series (Artetxe et al., 2021), the GPT-Neo(X)/J models (Black et al., 2021; Wang & Komatsuzaki, 2021; Black et al., 2022), the OPT series (Zhang et al., 2022), the BigScience Architecture and Scaling Pile model (Scao et al., 2022b), PaLM-8B (Chowdhery et al., 2022), Aleph Alpha Luminous…”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2021) Privacy Filtering private/duplicate data (Henderson et al., 2022; Kandpal et al., 2022; Lee et al., 2022b)”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “…on a copy of the Pile after applying near-deduplication with MinHashLSH and a threshold of 0.87, following the advice that LLMs trained on deduplicated data are better and memorize less of their data (Lee et al., 2021 modeling (Black et al., 2022; Chowdhery et al., 2022; Zeng et al., 2022): 1.”
Survey of Hallucination in Natural Language Generation (2022, background): “[96] show that duplicated examples from the pretraining corpus bias the model to favor generating repeats of the memorized phrases from the duplicated examples.”
Foundation Models and Fair Use (2023, background): “The fewer times a model sees an example during training, the less likely it will memorize it (Lee et al., 2021; Kandpal et al., 2022).”
Data Portraits: Recording Foundation Model Training Data (2023, background): “Observe that if a string of interest is at least of length 2 n − 1 it will necessarily contain at least one tile of size n , producing a match.”