Scaling Data-Constrained Language Models

By Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, Colin Raffel. Neural Information Processing Systems, 2023

422 citations recorded by the source corpus.

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/9e16d8cc6096ec0d2733a4ecf41ce09d9a4bd19c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Preference Ranking Optimization for Human Alignment (2023, background): “The proposed PRO instead distinguishes y i against all members from the sub-ranking y 1 , ··· ,n . et al., 2023; Li et al., 2023), primarily attributed to the extensive range of information sources integrated into their pretraining datasets (Laurençon et al., 2022; Muennighoff et al., 2023).”
OLMo: Accelerating the Science of Language Models (2024, methodology): “The impact of repeating this small amount of data should be negligible according to prior work (Muennighoff et al., 2023).”
C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, background): “The recent popularity of large language models (LLMs) has made text embeddings even more important.”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, methodology): “Following Muennighoff et al. (2023), we train for multiple epochs in each dataset and perform early stopping.”
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “We note that increasingly the number of tokens seen during pretraining has been incorporated in the study of scaling laws (Hoffmann et al., 2022; Muennighoff et al., 2023).”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, positive): “(Kaplan et al. 2020; Hoffmann et al. 2022) However, this approach is limited to the amount of available high-quality training data (Villalobos et al. 2024; Muennighoff et al. 2023).”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, background): “The impetus for this effort originates in the seemingly predictable improvement one obtains by training large models, the so-called scaling laws [KMH + 20, HBM + 22, MRB + 23].”
Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs (2025, unknown): “Over time these have converged, as exemplified by the MTEB benchmark (Muennighoff et al., 2023a), which incorporates BEIR (Thakur et al., 2021) as well as many other benchmarks for tasks such as semantic similarity.”