Energy and Policy Considerations for Deep Learning in NLP

By Emma Strubell, Ananya Ganesh, Andrew McCallum. 2019

3,520 citations recorded by the source corpus.

Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “As models get larger, improving training and inference efﬁciency become more important to keep training and inference times within reasonable limits from both a practical and environmental perspective (Strubell et al., 2019).”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “…et al., 2021), including bias in representations (Blodgett et al., 2020), proper handling (Rogers, 2021) and documentation (Bender & Friedman, 2018) of training data, privacy (Abadi et al., 2016b; Carlini et al., 2021), and environmental concerns (Strubell et al., 2019; Patterson et al., 2021).”
Improving language models by retrieving from trillions of tokens (2021, background): “In addition to the benefits of updating models in terms of fairness and bias, simply training large language models has a significant energy cost (Schwartz et al., 2020; Strubell et al., 2019).”
Ethical and social risks of harm from Language Models (2021, background): “This includes concerns about the working conditions of data annotators or “ghost workers” (Gray and Suri, 2019), the ethics of supply chains of hardware on which LM computations are run (Crawford, 2021), or environmental costs of training such models (Bender et al., 2021; Patterson et al., 2021; Sch”
Challenges in Detoxifying Language Models (2021, background): “, 2020), and environmental impact (Strubell et al., 2019),”
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, background): “While effective, it is also extremely computationally intensive (Strubell et al., 2019).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “We also note that few, if any, benchmark currently reports the environmental side-effects of training and serving NLG models (Strubell et al., 2019).”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “Other dimensions include energy consumption (both for development and testing) [Strubell et al., 2019, Henderson et al., 2020, Schwartz et al., 2020, Ethayarajh and Jurafsky, 2020]; memory and compute requirements, which may be more or less constrained depending on the deployment context…”
Deduplicating Training Data Makes Language Models Better (2021, background): “And so because these datasets are up to 19% smaller, even including the deduplication runtime itself, training on dedupli-cated datasets directly reduces the training cost in terms of time, dollar, and the environment (Bender et al., 2021; Strubell et al., 2019; Patterson et al., 2021).”
Rethinking Attention with Performers (2020, background): “Environment: As we have shown, Performers with FAVOR are characterized by much lower compute costs and substantially lower space complexity which can be directly translated to CO2 emission reduction (Strubell et al., 2019) and lower energy consumption (You et al.”
Big Bird: Transformers for Longer Sequences (2020, background): “Moreover, there is a growing concern in the ML community about the resource and energy requirement training large scale systems [81].”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Moreover, the computational costs have also lead to concerns about the carbon footprint stemming from the training and use of large language models (Strubell et al., 2019; Lacoste et al., 2019; Schwartz et al., 2020; Bannour et al., 2021), and existing carbon footprint studies have likely…”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “A significant point of concern in some recent work is the energy usage and carbon emissions associated with training large language models (Strubell et al., 2019; Schwartz et al., 2020; Lacoste et al., 2019; Bender et al., 2021).”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “Note, we present the operational carbon footprint of the OSS model training from (Strubell et al., 2019; Patterson et al., 2021).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, methodology): “[233] and examining environmental impact across compute-related impacts, immediate impacts of applying AI, and system-level impacts [120].”
The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, background): “Given that training a single moderately large model produces twice the American lifetime worth of CO2 [Strubell et al., 2019], we opted to not run such an experiment and instead focus on a more realistic setting for a proof-of-concept.”
Measuring the Carbon Intensity of AI in Cloud Instances (2022, methodology): “, [42]), or post-hoc analyses using information that is not publicly available (e.”