Carbon Emissions and Large Neural Network Training

By David A. Patterson, Joseph E. Gonzalez, Quoc V. Le, Chen Liang, Lluís-Miquel Munguía, Daniel Rothchild, David R. So, Maud Texier, Joseph Dean. 2021

1,164 citations recorded by the source corpus.

The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume<1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry, Efficiency & Compression, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Materials Chemistry

In Efficiency & Compression

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “For details on the distinction between conventional carbon offsets, Google’s goal for 2030 of 24/7 CFE for its global datacenters and campuses, and what it is doing now to set the groundwork for 2030, please see Appendix B of Patterson et al. (2021).”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Following preceding research (Bender et al., 2021a; Patterson et al., 2021; Wu et al., 2022; Dodge et al., 2022) and using power consumption estimates of GPU devices and carbon efficiency, we aim to calculate the carbon emissions resulting from the pretraining of Llama 2models.”
High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “This has two consequences for the research community and users in general: Firstly, training such a model requires massive computational resources only available to a small fraction of the field, and leaves a huge carbon footprint [63, 83].”
LaMDA: Language Models for Dialog Applications (2022, background): “The total FLOPS is 56.5% * 123 TFLOPS/s * 1024 chips * 57.7 days = 3.55E+23, which is higher than 3.14E+23, corresponding to the total FLOPS of GPT-3 [12].”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “There is also a concern about the environmental costs associated with training and deploying such models (Strubell et al., 2019; Gupta et al., 2021; Bender et al., 2021; Patterson et al., 2021) motivating research into more efficient model designs (Lepikhin et al.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Existing large dense language models usually require tremendous amounts of computation resources for training and serving (Patterson et al., 2021).”
Ethical and social risks of harm from Language Models (2021, background): “…most companies today spend more energy on operating deep neural network models (performing inference) than on training them: Amazon Web Services claimed that 90% of cloud ML demand is for inference, and Nvidia claimed that 80-90% of the total ML workload is for inference (Patterson et al., 2021).”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “Training Transformers incurs both financial and environmental costs (Schwartz et al., 2019; Patterson et al., 2021).”
Deduplicating Training Data Makes Language Models Better (2021, methodology): “We used data from Patterson et al. (2021) to estimate amount of energy used in training these models by computing the amount of MW h /hour/core and multiplying by our usage (see Table 6 for how we computed these values).”
Evaluating Large Language Models Trained on Code (2021, background): “Codex, like other large generative models, has an energy footprint from both training and inference (Schwartz et al., 2019; Bender et al., 2021; Patterson et al., 2021).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Both BLOOM and OPT incurred significantly less carbon emissions than GPT-3 (as reported by (Patterson et al., 2021)), which can be attributed to several factors including more efficient hardware as well as less carbon-intensive energy sources.”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “While OPT-175B was developed with an estimated carbon emissions footprint (CO2eq) of 75 tons, GPT-3 was estimated to use 500 tons (Patterson et al., 2021), while Gopher required 380 tons (Rae et al., 2021).”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “Although recent work shows the carbon footprint of training one large ML model, such as Meena (Patterson et al., 2021), is equivalent to 242,231 miles driven by an average passenger vehicle (EPA, 2021), this is only one aspect; to fully understand the real environmental impact we must consider the A”
Measuring the Carbon Intensity of AI in Cloud Instances (2022, methodology): “The estimates from Patterson et al. [34] range between 427 and 545 (except GShard 600B with 200), and Patterson et al. [33] found 88.”
A Review of Sparse Expert Models in Deep Learning (2022, background): “These models are therefore expensive to produce and incur high energy costs (Patterson et al., 2021).”
Predictability and Surprise in Large Generative Models (2022, background): “However, similarly to [72], we do not consider here the costs of human labor involved in creating and annotating training data [28], the ethics of supply chains involved in creating the requisite hardware on which to train models [18], or the environmental costs of training models [7, 50, 62, 66].”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “One of the major concerns about large language models is their huge energy usage and associated carbon emissions (Strubell et al., 2019; Lacoste et al., 2019; Patterson et al., 2021; Bender et al., 2021).”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “It is also important to consider the location where these TPUs are located, as the renewables portion at each datacenter is not equal [87].”