The Curse of Recursion: Training on Generated Data Makes Models Forget

By Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Nicolas Papernot, Ross Anderson. 2023

507 citations recorded by the source corpus.

Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Lastly, broader societal issues like job displacement due to accelerated AI research and an over-reliance on LLMs leading to training data degradation are also pertinent considerations (Acemoglu and Restrepo, 2018; Autor and Salomons, 2018; Webb, 2019; Shumailov et al., 2023).”
Reinforced Self-Training (ReST) for Language Modeling (2023, background): “The last term is therefore a form of offline policy gradients which prevents 𝜋 𝜃 ( 𝒚 | 𝒙 ) to move too far from 𝑝 ( 𝒚 | 𝒙 ) which could lead to model collapse (Shumailov et al., 2023).”
Scaling Synthetic Data Creation with 1,000,000,000 Personas (2024, background): “This can not only help deepen our understanding of complex systems but also speed up innovation by facilitating rapid iteration and experimentation.”
Self-Consuming Generative Models Go MAD (2023, background): “It was recently shown in related work [53] that the expected Wasserstein-2 metric E[dist(G,Pr)], or distributional distance, is increasing for this process.”
Will we run out of data? Limits of LLM scaling based on human-generated data (2022, background): “One challenge is that models might lose information about the original human data distribution, such that iteratively training on model outputs results in increasingly homogeneous and unrealistic outputs (Shumailov et al., 2023).”
Artificial intelligence and illusions of understanding in scientific research (2024, unknown): “This kind of systemic epistemic risk might be especially likely to accrue when new AI models are trained on the synthetic data outputs of previous models 146 .”
Tracking the perspectives of interacting language models (2024, methodology): “Our setting – where models are sequentially trained on each others outputs without intervention – can be viewed as a generalization of a single model sequentially trained on its own outputs as studied in (Shumailov et al., 2024).”
Gaussian mixture models as a proxy for interacting language models (2025, unknown): “We find that interacting GMMs can replicate the complex time varying dynamics, such as stable and unstable siloing, of interacting LLMs, but with a more easily interpretable underlying algorithm and far less computational cost.”
Understanding Generative AI Content with Embedding Models (2024, result): “This is further underscored by the results in [52] and [53, 54] which show that data produced by generative models are necessarily prone to hallucination and repeated training on such data which has been shifted away from the reference can lead to model collapse.”
LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs (2025, unknown): “Shumailov et al. (2024) first formalize this phenomenon as model collapse and demonstrates its existence in practice via GMMs, variational au-toencoders (VAEs), and LLMs. Quantifiable and interpretable model collapse have also attracted many attention in recent literature.”
LLM as a Broken Telephone: Iterative Generation Distorts Information (2025, unknown): “Iterative training on synthetically generated data induces model collapse, a phenomenon characterized by systematic erosion of the long-tail components of the original data distribution (Shumailov et al., 2023).”
When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (2024, unknown): “…2024; Marlow and Wood, 2021; Cabanac et al., 2021), journalism (Petridis et al., 2023), story generation (Valentini et al., 2023; Xie et al., 2023; Zhao et al., 2023), as chatbots on social media (Sadiko˘glu et al., 2023) and in the workplace at large (Brynjolfsson et al., 2023; Eloundou et al.,…”