AI models collapse when trained on recursively generated data
By Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal. Nature, 2024
891 citations recorded by the source corpus.
Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1), GPT-3(.5) (ref. 2) and GPT-4 (ref. 3) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet. Analysis shows that indiscriminately training generative artificial intelligence on real and generated content, usually done by scraping data from the Internet, can lead to a collapse in the ability of the models to generate diverse high-quality output.
View Paper: https://www.semanticscholar.org/paper/603d3f90fc40f79ff51258f0295de3ec5107f73e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Detecting Perspective Shifts in Multi-agent Systems (2025, unknown): “For a fixed t, t ′ query set Q , and embedding function g , we test whether a particular agent n changes between timepoints: Our proposed agent-level test ( TDKPS ) computes the Euclidean distance between TDKPS representations at time-points t and t ′ as the test statistic, || ψ i || F .”
- Understanding generative AI output with embedding models (2025, unknown): “This is further underscored by recent results showing that data produced by generative models are necessarily prone to hallucination ( 72 ) and that repeated training on such data can lead to model collapse ( 73 ).”
- Understanding Generative AI Content with Embedding Models (2024, result): “This is further underscored by the results in [52] and [53, 54] which show that data produced by generative models are necessarily prone to hallucination and repeated training on such data which has been shifted away from the reference can lead to model collapse.”
- When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (2024, unknown): “…of generative models trained on the preceding model’s output have been shown to sometimes collapse toward the most likely outputs while the tails of the original distribution disappear (Shumailov et al., 2024; Peterson, 2024; Gerstgrasser et al., 2024; Kazdan et al., 2024; Dohmatob et al., 2024).”
- Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models (2026, unknown): “There seems to be a general consensus that the unintentional ingestion of data generated by an LLM into other LLMs should eventually degrade performance, [31], [32], [33] but estimates of how serious this problem will become and when are in their infancy.”