Scaling Laws and Interpretability of Learning from Repeated Data

By Danny Hernandez, Tom B. Brown, Tom Conerly, Nova Dassarma, Dawn Drain, S. El-Showk, Nelson Elhage, Zac Hatfield-Dodds, T. Henighan, Tristan Hume and 8 more. arXiv.org, 2022

166 citations recorded by the source corpus.

Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or unintentionally because data deduplication is not perfect and the model is exposed to repeated data at the sentence, paragraph, or document level. Some works have reported substantial negative performance effects of this repeated data. In this paper we attempt to study repeated data systematically and to understand its effects mechanistically. To do this, we train a family of models where most of the data is unique but a small fraction of it is repeated many times. We find a strong double descent phenomenon, in which repeated data can lead test loss to increase midway through training. A predictable range of repetition frequency leads to surprisingly severe degradation in performance. For instance, performance of an 800M parameter model can be degraded to that of a 2x smaller model (400M params) by repeating 0.1% of the data 100 times, despite the other 90% of the training tokens remaining unique. We suspect there is a range in the middle where the data can be memorized and doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs. Finally, we connect these observations to recent mechanistic interpretability work - attempting to reverse engineer the detailed computations performed by the model - by showing that data repetition disproportionately damages copying and internal structures associated with generalization, such as induction heads, providing a possible mechanism for the shift from generalization to memorization. Taken together, these results provide a hypothesis for why repeating a relatively small fraction of data in large language models could lead to disproportionately large harms to performance.

Subfields: Optimization & Training, Scaling & Learning Theory

View Paper: https://arxiv.org/abs/2205.10487

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “2022) also show that LM perplexity is harmed by data duplication, but only for an intermediate amount of duplication.”
Galactica: A Large Language Model for Science (2022, neutral): “This is unexpected as existing research suggests repeated tokens can be harmful on performance (Hernandez et al., 2022).”
The Stack: 3 TB of permissively licensed source code (2022, background): “Hernandez et al. (2022) also found that repeating even a small portion of the training data can significantly hurt model performance, potentially due to a fraction of its capacity being consumed by data memorization.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “Secondly, we did not want to over-index on scaling studies that only measure impact on upstream cross entropy (Hernandez et al., 2022) which claims that repeating data in small quantities could be dis-proportionally harmful.”
Scaling Data-Constrained Language Models (2023, background): “Using multiple epochs is, of course, standard in machine learning generally; however, most prior large language models have been trained for a single epoch [51, 15] and some work explicitly advocates against reusing data [40].”
Explainability for Large Language Models: A Survey (2023, background): “Extensive experiments on multiple medical image datasets demonstrate the effectiveness and explainability of this training-free diagnostic pipeline.”
Large Language Models: A Survey (2024, unknown): “Some works [125] have shown that de-duplication improves models’ ability to generalize to new, unseen data.”
Data Selection for Language Models via Importance Resampling (2023, unknown): “[26]showthatdeduplicatingdataimprovesLMs, and Baevskietal.”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Jin et al. [121] conducted a comprehensive study on LLMs’ understanding of binary code semantics [122] across different architectures and optimization levels, providing key insights for future research in this area.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “The ultimate goal is to uncover general principles that transfer across models and tasks, such as induction heads for in-context learning (Olsson et al., 2022), variable binding mechanisms (Feng & Steinhardt, 2023; Davies et al., 2023), arithmetic reasoning (Stolfo et al., 2023; Brinkmann et al.,…”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Such duplication can shift LLMs from generalization to memorization (Hernandez et al., 2022), ultimately giving rise to a duplication bias where LLMs over-prioritize the recall of duplicated data and lead to hallucinations that deviate from the desired content.”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, unknown): “Recent studies have increasingly highlighted the detrimental effects of low-quality and repeated content in pre-training corpora on model training, as evidenced in various works [1, 42, 33, 27, 41].”
Emergent and Predictable Memorization in Large Language Models (2023, background): “In the overwhelmingmajority of prior work on scaling laws [7, 8, 15, 32, 37, 41, 43, 47], including scaling studies targeting memorization [13, 25, 51], plots of quantities of interest vs compute are linear on a log or log-log plot.”
The MiniPile Challenge for Data-Efficient Language Models (2023, background): “Potential issues with such include token repetitions [33, 21, 1], misogyny, pornography, and malignant stereotypes [9, 7], benchmark data contamination [11, 14], spurious correlations [27, 37, 38], diluted robustness due to data mixing [45] and potentially sensitive information [20].”
A Survey of Large Language Models (2023, unknown): “Besides, it has been shown that duplicate data degrades the ability of LLMs to copy from the context, which might further affect the generalization capacity of LLMs using in-context learning [146].”
Training Trajectories of Language Models Across Scales (2022, methodology): “…parameters (Liu et al., 2021; Choshen et al., 2022; Blevins et al., 2022) or focuses on the language modeling objective alone (Kaplan et al., 2020; Hernandez et al., 2021, 2022), we are the first to comprehensively study the training trajectories of large-scale autoregressive language models with…”