Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model

By Alexandra Sasha Luccioni, S. Viguier, Anne-Laure Ligozat. 2022

557 citations recorded by the source corpus.

Progress in machine learning (ML) comes with a cost to the environment, given that training ML models requires significant computational resources, energy and materials. In the present article, we aim to quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes of~\carboneq~if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also study the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of ML models and future research directions that can contribute towards improving carbon emissions reporting.

Subfields: Electrical and Electronic Engineering, Materials Chemistry, Large Language Models, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

1. Irene Solaiman
2. Alexandra Sasha Luccioni
3. Sara Hooker

In Large Language Models

1. Irene Solaiman
2. Zeerak Talat
3. Yacine Jernite
4. Alexandra Sasha Luccioni
5. Jesse Dodge
6. Margaret Mitchell

In AI Ethics, Fairness & Society

1. Alexandra Sasha Luccioni
2. Irene Solaiman
3. Marie-Therese Png
4. Zeerak Talat
5. Yacine Jernite
6. Su Lin Blodgett
7. Margaret Mitchell
8. Sara Hooker

Citations in the corpus

• Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Scope 3 emissions, the indirect emissions often outside a developer’s control, should account for a generative AI system’s lifecycle including in deployment [141].”
• BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024, methodology): “According to documentation from the Jean Zay supercomputer 4 , the total environmental cost amounted to 1,295,000 Wh or 73.8 kg CO2eq, based on the carbon intensity of the energy grid as reported in the BLOOM environmental cost study conducted on Jean Zay (Luccioni et al., 2022).”
• Power Hungry Processing: Watts Driving the Cost of AI Deployment? (2023, unknown): “While the majority of existing work has been focused on ML model training given that it is a more tractable part of the model life cycle (i.e. it is most often carried out over a set period of time on a specific compute instance), model inference has started to also become the subject of…”
• Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, background): “For example, the training of GPT-3 is estimated to have consumed 1287 MWh [72]—approximately the same energy as 64 Swedish homes [7].”