The Cost of Training NLP Models: A Concise Overview

By Or Sharir, Barak Peleg, Yoav Shoham. 2020

238 citations recorded by the source corpus.

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, methodology): “Other variables used to calculate cost include size of dataset, model size, and training volume [218].”
The Efficiency Misnomer (2021, background): “One can also consider the expenses for training or serving models as a cost indicator which is reported in the form of figures (Sharir et al., 2020).”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “Despite their impressive performance, however, their often massive computational burden makes them costly to run (Schwartz et al., 2019; Sharir et al., 2020).”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, background): “The carbon footprint of retraining such systems for every new data update is considerable, and will likely grow in the coming years [17].”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, background): “Due to the high cost of training large, state-of-the-art transformer models [45, 5], methodologies that allow one to build off of previously trained models cheaply are highly valuable.”
Dynamic Language Models for Continuously Evolving Content (2021, methodology): “However, simply using all new data can be very costly, as training BERT is computationally expensive [38].”
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training (2021, background): “Massive training datasets have significantly increased end-to-end training times, computational and resource costs (Sharir et al., 2020), energy requirements (Strubell et al., 2019), and carbon footprint (Schwartz et al., 2019).”
On the Difficulty of Defending Self-Supervised Learning against Model Extraction (2022, background): “High-performance SSL models in these domains are often costly to train; training a large BERT model can cost north of 1 million USD (Sharir et al., 2020).”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “As neural networks have gotten bigger [6, 7] and more expensive to run [8, 9], it has become more pressing to ﬁnd ways to address this waste.”
Limits to Depth Efficiencies of Self-Attention (2020, background): “Such insights may prove useful given the rapid increase in model sizes [Sharir et al., 2020].”
ASPEN: High-Throughput LoRA Fine-Tuning of Large Language Models with a Single GPU (2023, unknown): “It can take several days using thousands of GPUs [33] and entail significant financial investment [39].”
Tutel: Adaptive Mixture-of-Experts at Scale (2022, background): “However, DNN model capacity is often limited by computing resource and energy cost [39].”
Confident Adaptive Language Modeling (2022, background): “The large size of LLMs, however, results in massive computational load that might be limiting for certain real-world applications (e.g., machine translation) [9; 30; 42; 49; 59; 63; 71].”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “LLMs are characterized by fixed parametric knowledge, a result of the substantial pertaining process (Sharir et al., 2020; Hoffmann et al., 2022; Smith, 2023).”
Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives (2022, background): “In recent NLP studies, it is a standard training strategy to fine-tune a transformer-based model with a small amount of data for a target application.”
Shifting machine learning for healthcare from development to deployment and from models to data (2022, background): “Because transformers and deep NLP models are resource-intensive to train (training the BERT model can cost US$50,000–200,000 133 ; and pre-training BERT on clinical datasets required 18 d of continuous training, an endeavour that may be out of the reach of many institutions), openly releasing…”
Compute Trends Across Three Eras of Machine Learning (2022, background): “For example, Sharir et al. (2020) estimates that Google’s T5 project—which is by no means the biggest training run to date—might have cost a total of $10M in cloud computing costs.”