Self-Consuming Generative Models Go MAD
By Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, Richard Baraniuk. International Conference on Learning Representations, 2023
319 citations recorded by the source corpus.
Seismic advances in generative AI algorithms have led to the temptation to use AI-synthesized data to train next-generation models. Repeating this process creates autophagous (“self-consuming”) loops whose properties are poorly understood. We conduct a thorough analysis using state-of-the-art generative image models of three autophagous loop families that differ in how they incorporate fixed or fresh real training data and whether previous generations' samples have been biased to trade off data quality versus diversity. Our primary conclusion across all scenarios is that without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or diversity (recall) progressively decrease. We term this condition Model Autophagy Disorder (MAD) and show that appreciable MADness arises in just a few generations.
Subfields: Management Information Systems
View Paper: https://www.semanticscholar.org/paper/ad022cd5da75637ac3e0a8f8cc4f0d394ba5ff7a
Citations in the corpus
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “However, the information from the Internet is in the wild, which means they may also be fabricated, or even generated by LLMs themselves (Alemohammad et al., 2023).”
- LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs (2025, unknown): “Alemohammad et al. (2023) introduce self-consuming training loops and show that, without a constant infusion of fresh real data, models converge to “Model Autophagy Dis-order” (MAD).”
- Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models (2026, unknown): “There seems to be a general consensus that the unintentional ingestion of data generated by an LLM into other LLMs should eventually degrade performance, [31], [32], [33] but estimates of how serious this problem will become and when are in their infancy.”