Compute Trends Across Three Eras of Machine Learning

By Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos. 2022

431 citations recorded by the source corpus.

Compute, data, and algorithmic advances are the three fundamental factors that drive progress in modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We make three novel contributions: (1) we curate a dataset with the training compute of 123 milestone ML systems, 3× larger than previous such datasets. (2) We frame the trends in compute in in three eras - the Pre Deep Learning Era, the Deep Learning Era, and the Large-Scale Era, based on our identification of a novel trend emerging around 2015. (3) We find a Deep Learning Era compute doubling time of around 6 months, significantly longer than previous findings. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.

Subfields: Artificial Intelligence, Scaling & Learning Theory, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

In Evaluation & Benchmarks

Citations in the corpus

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Progress in natural language processing is increasingly driven by sheer compute scale alone (Sevilla et al., 2022): as more compute is expended to train large language models (LLM), they gain and exhibit powerful emergent capabilities (Brown et al.”
Will we run out of data? Limits of LLM scaling based on human-generated data (2022, methodology): “For this purpose, we use the results from Besiroglu et al. (2022), which performs such a projection based on estimated training compute growth rates in frontier machine learning systems between 2010 and 2022.”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “Scale is a major force behind the accuracy improvements of machine-learning-based solutions [9], and both the depth and width of deep neural networks (DNN) are expanding exponentially [66] (Figure 1).”
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference (2023, background): “While the complete details (time and resources used) of compute required for training GPT-3/4 are not available, several estimates for training [10], [11] and inference are publicly available.”
Auditing large language models: a three-layered approach (2023, background): “Further, independent auditors may lack the computational resources to conduct audits that go beyond interrogating relationships between prompts and outputs [121].”