GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

By Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat and 16 more. 2021

1,261 citations recorded by the source corpus.

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “…computation large MoE models like the 1.7 trillion parameter Switch transformer (Fedus et al., 2021), the 1.2 Trillion parameter GLaM model (Du et al., 2021), and others (Artetxe et al., 2021; Zoph et al., 2022) are able to provide a large eﬀective model size despite using relatively fewer…”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “For instance, in Table 4 we present a new few-shot state of the art on 28 out of the 29 most widely evaluated English language understanding benchmarks, compared against the best per-task result from GLaM (Du et al., 2021), GPT-3 (Brown et al.”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “, 2020) and GLaM 64B/64E (Du et al., 2021), as reported in their respective papers.”
GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, background): “Recent research has shown the benefit of scaling up the Transformer model [1, 8, 13, 18, 33] by increasing the number of layers and the sizes of dimensions M, H and N, which could reach hundreds of billions of parameters.”
Emergent Abilities of Large Language Models (2022, neutral): “For example, Chinchilla (Hoffmann et al., 2022) has one-fourth as many parameters as Gopher (Rae et al., 2021) but uses similar training compute; and sparse mixture-of-expert models have more parameters per training/inference compute than dense models (Fedus et al., 2021; Du et al., 2021).”
Scaling Instruction-Finetuned Language Models (2022, methodology): “, 2022), which did not find any meaningful data contamination with pre-training data, consistent with data contamination analyses in previous work (Brown et al., 2020; Wei et al., 2021; Du et al., 2022).”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “As a result, the MoE block uses a small portion of all “experts” for any single forward pass, allowing for more compute-efficient training Fedus et al. (2021); Du et al. (2022).”
A Review of Sparse Expert Models in Deep Learning (2022, methodology): “The state-of-the-art sparse models for few-shot inference (GLaM (Du et al., 2021)) and for fine-tuning (ST-MoE (Zoph et al.”
Mixture-of-Experts with Expert Choice Routing (2022, methodology): “Model Training: Our model training follows the setups of GLaM [7] where a maximum sequence length of 1024 tokens is adopted.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “However, so far, these models have shown limited performance on open-text generation and prompt-based inference (i.e., in-context learning), which motivates the use of decoderonly models that are trained with different objectives (e.g., GPT-3 (Brown et al., 2020), GLaM (Du et al., 2021), LaMDa (Thop”
Predictability and Surprise in Large Generative Models (2022, background): “, GLaM [22]), which often achieve comparable performance with similar or slightly lower compute, but are difficult to characterize in terms of a single model size.”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, background): “It proves to be effective for various convolutional neural works (CNNs) (Han et al., 2016; Jacob et al., 2018; Nagel et al., 2019; Wang et al., 2019; Lin et al., 2020) and transformers (Shen et al., 2020; Kim et al., 2021; Liu et al., 2021; Wang et al., 2020; Bondarenko et al., 2021).”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “As such, there has been an explosion of work on developing LLMs and training techniques that produce strong zero-shot generalization [Brown et al., 2020, Wang and Komatsuzaki, 2021, Du et al., 2021, Lin et al., 2021, Rae et al., 2021, Hoffmann et al., 2022, Chowdhery et al., 2022].”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Even so, autoregressive models for text-to-image generation remain appealing given extensive prior work on scaling large language models [17, 18, 19, 20] and advances in discretizing other modalities–such as images and audio–so that inputs in those modalities can be treated as language-like tokens.”