What Language Model to Train if You Have One Million GPU Hours?

By Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, Saiful Bari, Stella Biderman, Hady ElSahar, Niklas Muennighoff, Jason Phang and 9 more. 2022

125 citations recorded by the source corpus.

The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations can transfer across tasks and scale, increasing the impact of modeling research. However, with the emergence of state-of-the-art 100B+ parameters models, large language models are increasingly expensive to accurately design and train. Notably, it can be difficult to evaluate how modeling decisions may impact emergent capabilities, given that these capabilities arise mainly from sheer scale alone. In the process of building BLOOM--the Big Science Large Open-science Open-access Multilingual language model--our goal is to identify an architecture and training setup that makes the best use of our 1,000,000 A100-GPU-hours budget. Specifically, we perform an ablation study at the billion-parameter scale comparing different modeling practices and their impact on zero-shot generalization. In addition, we study the impact of various popular pre-training corpora on zero-shot generalization. We also study the performance of a multilingual model and how it compares to the English-only one. Finally, we consider the scaling behaviour of Transformers to choose the target model size, shape, and training setup. All our models and code are open-sourced at https://huggingface.co/bigscience .

Subfields: Artificial Intelligence, Hardware and Architecture, Materials Chemistry, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Materials Chemistry

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “See Le Scao et al. (2022) for a detailed list of tasks and baselines.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “We find C4 to be a strong pretraining dataset, in line with the findings of Scao et al. (2022b)–however, The Pile comparatively underperforms more in our benchmarks.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “ROOTS (Laurenc¸on et al., 2022), the dataset that BLOOM (Scao et al., 2022) was trained on, was styled after the Pile and would potentially be a good candidate, but it was not publicly available when we started training our models.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Another option from BLOOM-176B is to apply embedding norm with BF16, but in sacrifice of a significant penalty on model performance (Scao et al., 2022).”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “…generalist webcrawls [Ortiz Suárez et al., 2019, Raffel et al., 2020], collected with no explicit regard for downstream tasks–although adding curated high-quality cross-domain data has been proposed as a path towards better zero-shot generalization [Gao et al., 2020, Scao et al., 2022].”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, methodology): “ALiBi , used in BLOOM (Scao et al., 2022a), is similar to T5’s Relative Bias but instead subtracts a scalar bias from the attention score.”
When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “For one thing, although there are no speciﬁc criteria for the minimum number, a typical LLM usually consists of no less than severalbillionanduptotrillionsofmodelparameters,whichareorderslargerthanbefore[33].”
Emergent and Predictable Memorization in Large Language Models (2023, background): “…of a variety of researchers at model training-time by allowing them to generalize the decisions made while investigating smaller models to inform the design of larger (sometimes by many orders of magnitude) models [Rae et al., 2021, Black et al., 2022, Scao et al., 2022b, Chowdhery et al., 2022].”
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model (2022, unknown): “We present some key numbers about BLOOM model training in Table 1 below, and refer readers to [5, 19] for additional information about model architecture and training.”
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster (2023, background): “Another notable work that releases dataset and model is the Big Science collaborative eﬀort to train BLOOM 176B (Scao et al., 2022; 2023).”
Training Trajectories of Language Models Across Scales (2022, methodology): “When comparing models trained with different corpora, such as Neo GPT NEO (Black et al., 2021) and BLOOM (Scao et al., 2022), different architectures and objectives, such as retrieval-based language models (Khandelwal et al., 2020; Zhong et al., 2022; Borgeaud et al., 2021) and sparse models (Fedus ”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “This burgeoning class of LLMs, underpinned by increasingly sophisticated architectures and training methodologies [8, 48], has the potential to revolutionize various domains by offering unprecedented capabilities in natural language understanding and generation.”