An empirical analysis of compute-optimal large language model training

By Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark and 12 more. 2022

516 citations recorded by the source corpus.

Subfields: Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “NLG Evaluation in times of large language models Over the last years, research labs started developing releasing large language models (Brown et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2022).”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “Most language models [25, 26, 24, 21, 27, 22] use byte-pair encoding (or equivalent) as a trade-off between character-level and word-level tokenization.”
Explaining neural scaling laws (2021, background): “Further let f T be concentrating with Case 1 – ﬁnite degree polynomial : In this case, we can write, where K is the polynomial degree and ( k ) is the k -th derivative of ‘ .”
Data Selection for Language Models via Importance Resampling (2023, background): “Given a fixed compute budget, the choice of pretraining data is critical for the performance of language models (LMs) [18; 10; 24; 64; 27].”
Language Models can Solve Computer Tasks (2023, methodology): “[11] used Chinchilla[20] as a planner for an agent in the PycoLab environment, but their actor module requires pre-training with reinforcement learning (RL) to follow natural language instructions.”
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster (2023, methodology): “Language models can also be improved just by training on more data (Hoﬀmann et al., 2022; Touvron et al., 2023).”
Leveraging large language models for predictive chemistry (2024, unknown): “This works particularly well with the largest GPT-3 models and GPT-4.”
Proving Test Set Contamination in Black Box Language Models (2023, background): “Data contamination has been studied in many contexts, including in the study of pretraining corpora ((Dodge et al., 2021)) as well as in the analysis section of many language model papers (Hoffmann et al., 2022; Brown et al., 2020a; Gao et al., 2020).”
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023, background): “For example, The Pile [17], a large publicly available dataset, is composed of 24% web data, 9% Wikipedia, 4% GitHub, etc. 1 The composition of the pretraining data greatly affects the effectiveness of an LM [13; 20; 55].”
OpenFold: retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization (2023, unknown): “Our data elision results can be interpreted in light of recent work on large transformer-based language models that has revealed broadly applicable ‘scaling laws’ that predict model accuracy as a simple function of model size, compute used and training set size 47,48 .”
Health system-scale language models are all-purpose prediction engines (2023, background): “…Nature Portfolio reporting summaries, source data, extended data, supplementary information, acknowl-edgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at https://doi.org/10.1038/s41586-023-06160-y.”
AI and Memory Wall (2024, unknown): “Now, if we study the trend of recent AI models, and in particular LLMs, we notice that practitioners, motivated by neural scaling law [14], have been scaling the amount of data, model size, and compute needed to train recent models at unprecedented levels.”
Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “These models are trained to maximize the likelihood of the next token autoregressively using massive amounts of text and compute (Hoffmann et al., 2022; Srivastava et al., 2022).”
Foundation models for generalist medical artificial intelligence (2023, background): “One recent study established a link between dataset size and model size, recommending 20 times more tokens than parameters for optimal performance, yet existing foundation models were successfully trained with a lower token-to-parameter ratio 51 .”
BloombergGPT: A Large Language Model for Finance (2023, methodology): “From Hoffmann et al. (2022), we use the data reported in Table 3 for Approach 1 and Table A3 for Approach 2, and fit regression lines to their log-scaled versions.”