Green AI

By Roy Schwartz, Jesse Dodge, Noah A. Smith, Oren Etzioni. 2019

1,782 citations recorded by the source corpus.

The computations required for deep learning research have been doubling every few months, resulting in an estimated 300,000x increase from 2012 to 2018 [2]. These computations have a surprisingly large carbon footprint [38]. Ironically, deep learning was inspired by the human brain, which is remarkably energy efficient. Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers, in particular those from emerging economies, to engage in deep learning research. This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or"price tag"of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. Our goal is to make AI both greener and more inclusive---enabling any inspired undergraduate with a laptop to write high-quality research papers. Green AI is an emerging focus at the Allen Institute for AI.

Subfields: Computer Science Applications, Electrical and Electronic Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Electrical and Electronic Engineering

In Efficiency & Compression

Citations in the corpus

How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “If efficiency is a concern, however, such properties can be optimized directly by modifying pretraining-agnostic benchmarks to take them into account (Schwartz et al., 2019).”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “Moreover, although reporting FLOPs is useful for comparison as it is hardware-agnostic, it neglects the fact that parallel operations are signiﬁcantly cheaper than sequential operations on modern hardware.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “1https://github.com/pytorch/fairseq/ tree/main/examples/moe_lm Unfortunately, the corresponding growth in computational resources required to train state-of-theart language models is a barrier for many in the research community (Schwartz et al., 2019).”
Improving language models by retrieving from trillions of tokens (2021, background): “In addition to the beneﬁts of updating models in terms of fairness and bias, simply training large language models has a signiﬁcant energy cost (Schwartz et al., 2020; Strubell et al., 2019).”
Carbon Emissions and Large Neural Network Training (2021, background): “We agree with [Str19,Sch20,Hen20] that efficiency should be an evaluation criterion for publishing ML research on computationally intensive models besides accuracy and related measures, since we need to encourage advances across the board as the most sustainable energy is the energy you don’t use .”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “A significant point of concern in some recent work is the energy usage and carbon emissions associated with training large language models (Strubell et al., 2019; Schwartz et al., 2020; Lacoste et al., 2019; Bender et al., 2021).”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “There are several factors that may have contributed to the current state of AI: • Lack of incentives: Over 90% of the ML publications only focus on model accuracy improvements at the expense of efﬁciency [109].”
Chasing Carbon: The Elusive Environmental Footprint of Computing (2020, background): “The parameters of the energy footprint for training AI models are threefold: the footprint of processing one example ( E ), the data-set size ( D ), and the hyperparameter search ( H ) [15], [78].”
Mixture-of-Experts with Expert Choice Routing (2022, background): “However, training efﬁciency, which is deﬁned as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “compute required for few-shot learning is closely related to other efforts in Green AI (Schwartz et al., 2020a) that aim to improve model efficiency, including techniques for knowledge distillation (e.”
Energy and Policy Considerations for Modern Deep Learning Research (2020, background): “See Schwartz et al. (2019) for more discussion of standard metrics for reporting efﬁciency, and Dodge et al. (2019) for further analysis and concrete methods for reporting tuning and hyperparameter sensitivity.”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, background): “Environmental Impact The use of large-scale Transformers requires a lot of computations and GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019; Schwartz et al., 2020).”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “Data Efficiency & Evaluation Other researchers have noted the importance of cost-benefit trade-offs in evaluation (Schwartz et al. 2019).”
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “This concern is based in part on trends in computation for machine learning at large scale, as Schwartz et al. (2020) note a 300,000x increase in computation over 6 years leading up to 2019.”
Predictability and Surprise in Large Generative Models (2022, background): “However, similarly to [83], we do not consider here the costs of human labor involved in creating and annotating training data [30], the ethics of supply chains involved in creating the requisite hardware on which to train models [19], or the environmental costs of training models [7, 56, 69, 74].”
Choose Your Weapon: Survival Strategies for Depressed AI Academics (2023, background): “You can also argue for small models from the perspective of green AI [20] , as it minimizes the environmental footprint of the research.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “The large increase in the size of artificial intelligence models often increases production costs (Amodei and Hernandez, 2018; Schwartz et al., 2019), and can also limit adoption on real-time devices.”
Show Your Work: Improved Reporting of Experimental Results (2019, background): “But, as the ﬁeld grows, there is an increasing gap between the large computational budgets used for some high-proﬁle experiments and the budgets used in most other work (Schwartz et al., 2019).”