BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

By Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili'c, Daniel Hesslow, Roman Castagn'e, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé and 378 more. 2022

2,975 citations recorded by the source corpus.

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

Subfields: Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, background): “In the past two decades, neural networks have been successfully applied to the language modelling task, et al. (2022) to compute carbon emission of train OPT, BLOOM and our models in the same data center.”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Open-source releases like BLOOM (Scao et al., 2022) and Falcon (Penedo et al., 2023) have risen to challenge their closed-source counterparts like GPT-3 and Chinchilla.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “Latest large language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Scao et al., 2022a) are trained on giant aggregated corpora, combining both massive web-scrape and so-called “high-quality” curated single-domain sources (e.g., news, books, technical papers, social media…”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “However, the rapid scientific progress in this area would be impossible without open-access LLMs such as LLaMA 1 and 2 (Touvron et al., 2023), Falcon (TII UAE, 2023), BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022), or NeoX/Pythia (Biderman et al., 2023).”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “To additionally study the advantages of NF4 over other 4-bit data types, we use the setup of Dettmers and Zettlemoyer [13] and measure post-quantization zero-shot accuracy and perplexity across different models (OPT [73], LLaMA [58], BLOOM [53], Pythia [7]) for model sizes 125m - 13B.”
StarCoder: may the source be with you! (2023, background): “One example is the BigScience research workshop (BigScience Workshop, 2022), an open scientiﬁc collaboration (Akiki et al., 2022) comprising hundreds of researchers collaborating to release BLOOM, a multi-lingual LLM (Scao et al., 2022; Muennigho � et al., 2022).”
Stable and low-precision training for large-scale vision-language models (2023, background): “Another instance of float16 creating issues at scale was the training of OPT [74] and BLOOM models [52].”
Instruction Tuning with GPT-4 (2023, background): “Early attempts on foundation LLMs include BLOOM (Scao et al., 2022), GPT-J (Wang & Komatsuzaki, 2021), GPT-NEO (Black et al.”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “Large Language Models (LLMs) are widely adopted for zero/few-shot inference (Zhang et al., 2022; Black et al., 2022; Zeng et al., 2022; Scao et al., 2022), but they can be challenging to use both due to their large memory footprints – up to 352 GB of GPU memory for 175B models – and high latency.”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “, 2022), BLOOM (Scao et al., 2022), and GLM-130B (Zeng et al.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Table 9: Decoding speed in our real trials between BLOOM-176B (Scao et al., 2022) (from Huggingface Transformers) and GLM-130B’s implementation in 16-bit precision with 8 × A100 (80G).”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “In recent years, the development of Large language models (LLMs) has shown incredible progress, such as T5 [32], BLOOM [36], and GPT-3 [5].”
SantaCoder: don't reach for the stars! (2023, methodology): “BigCode was inspired by the BigScience project, an open-scientiﬁc collaboration which culminated in July 2022 with the release of a large multi-lingual language model (Scao et al., 2022).”
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, methodology): “, 2022) is based on the pretrained multilingual BLOOM model (Scao et al., 2022) after multitask prompted finetuning, thus is suitable for non-English languages.”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, positive): “To overcome this limitation, previous efforts have employed model compression techniques with specific designs to achieve efficient LLM inference, such as the method described in (Frantar & Alistarh, 2023), which employs one-shot pruning on LLMs (e.g., OPT-175 B and BLOOM-176 B), resulting in neglig”