Will we run out of data? Limits of LLM scaling based on human-generated data

By Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, Anson Ho. 2022

248 citations recorded by the source corpus.

We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress.

Subfields: Artificial Intelligence, Natural Language Processing, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Scaling & Learning Theory

Citations in the corpus

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “…scaling paradigm raises significant challenges: although plentiful, text data is not infinite, especially so when considerations on data quality and licensing are taken into account–leading some researchers to argue scaling may soon be bottlenecked by data availability (Villalobos et al., 2022).”
Pretraining Language Models with Human Preferences (2023, neutral): “C L ] 1 6 Fe b 20 23 bottlenecked by high-quality data (Hoffmann et al., 2022; Villalobos et al., 2022).”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, unknown): “(Kaplan et al. 2020; Hoffmann et al. 2022) However, this approach is limited to the amount of available high-quality training data (Villalobos et al. 2024; Muennighoff et al. 2023).”
Large language models in medicine (2023, methodology): “Where LLMs are used as tools, issues of responsibility and credit must be addressed 96,101–103 .”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “Previous study [88] suggests that high-quality language data is likely exhausted before 2026, and low-quality language and image data could be run out by 2060.”
Scaling Data-Constrained Language Models (2023, background): “This motivates the question [112, 81]: what should we do when we run out of data?”
Scaling Synthetic Data Creation with 1,000,000,000 Personas (2024, background): “This can not only help deepen our understanding of complex systems but also speed up innovation by facilitating rapid iteration and experimentation.”
LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs (2025, unknown): “On the one hand, neural-scaling law assumes model performance will improve as the amount of training data increases (Ka-plan et al., 2020); on the other hand, the pool of human-generated data is approaching its natural limit (Villalobos et al., 2024).”