Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
By William Fedus, Barret Zoph, Noam Shazeer. Journal of machine learning research, 2021
4,336 citations recorded by the source corpus.
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the"Colossal Clean Crawled Corpus"and achieve a 4x speedup over the T5-XXL model.
Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression
View Paper: https://arxiv.org/abs/2101.03961
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Yoshua Bengio
- Yejin Choi
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Furu Wei
- Jianfeng Gao
- Noah A. Smith
- Timo Schick
- Graham Neubig
In Efficiency & Compression
- Dan Alistarh
- Elias Frantar
- Luke S. Zettlemoyer
- Barret Zoph
- Quoc V. Le
- Alexander M. Rush
- Joseph Dean
- Tim Dettmers
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “These include both dense transformer models [6, 30, 48, 38, 52] and mixture-of-expert (MoE) models [11, 12, 60].”
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “, 2022) as well as trillion parameter sparse models including Switch Transformers (Fedus et al., 2021) and GLaM (Du et al.”
- Training language models to follow instructions with human feedback (2022, methodology): “…intervals. used for many recent large LMs—predicting the next token on a webpage from the internet—is different from the objective “follow the user’s instructions helpfully and safely” (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022).”
- Unified Scaling Laws for Routed Language Models (2022, methodology): “Routing was popularized for large scale training by Shazeer et al. [2017], and furthered by work including GShard [Lepikhin et al., 2020], Switch Transformer [Fedus et al., 2021] and GLaM [Du et al., 2021].”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “MoE allows for extreme scaling in terms of model sizes, with recent work reaching 1.6, 1.75 and even 10 trillion [5, 15, 36] parameters.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “In Table 7 we report training time and energy usage for our models based on the above estimates, as well as estimates from Patterson et al. (2021) for other large-scale LMs, in particular GShard (Lepikhin et al., 2021), Switch Transformer (Fedus et al., 2021) and GPT-3 (Brown et al., 2020).”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “2T parameters in total with 64 experts per MoE layer (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2021) where each token in the input batch only activates a subnetwork of 95B (8% of 1.”
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “There have been increasing demand for training and scaling Transformers (Shoeybi et al., 2019; Raffel et al., 2019; Fedus et al., 2021; Conneau et al., 2019; Lin et al., 2021).”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Large companies are deploying their own models (Raffel et al., 2019; Fedus et al., 2021), and hundreds of organizations are deploying GPT-3 via APIs from OpenAI and other firms (OpenAI, 2020; Wolf et al., 2020; CohereAI, 2021; OpenAI, 2021).”
- Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “…pipelined (Narayanan et al., 2021; Rasley et al., 2020; Narayanan et al., 2019; Huang et al., 2019) and homogeneous MoE (Lepikhin et al., 2020; Fedus et al., 2021) models on MPI-style systems, but as we argue in detail later, the MPI programming model is too restrictive both for users and for…”
- Quantifying Memorization Across Neural Language Models (2022, background): “The performance of neural language models has continuously improved as these models have grown from millions to trillions of parameters (Fedus et al., 2021), with their training sets similarly growing from millions to trillions of tokens.”
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “In addition to the traditional “dense” models, sparse mixture of experts (MoE) models of up to more than 1T parameters have been trained and finetuned (Lepikhin et al., 2020; Fedus et al., 2021).”
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “, 2021), as well as sparse computation (Fedus et al., 2021) can help address latency issues, removing the remaining barriers to a token-free future.”
- Carbon Emissions and Large Neural Network Training (2021, background): “…mixture-of-expert-style models can provide more than 10X reductions in computation requirements and energy costs for both training and inference while providing significantly higher accuracy than dense Transformer or LSTM-based models of equivalent computational cost per token [Sha17,Lep20,Fed21].”
- Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “1 C4 has been used to train models such as T5 and the Switch Transformer (Fedus et al., 2021), two of the largest pretrained English language models.”
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, methodology): “Switch Transformers [15] used Mesh-Tensorflow to train a sparsely activated expert-based model with 1.”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Both of these approaches are promising, and have now demonstrated competitive results–at large scales for MoE (Fedus et al., 2022; Srivastava et al., 2022), and at smaller scale for state-space models with H3 (Anonymous, 2023).”
- Emergent Abilities of Large Language Models (2022, neutral): “For example, Chinchilla (Hoffmann et al., 2022) has one-fourth as many parameters as Gopher (Rae et al., 2021) but uses similar training compute; and sparse mixture-of-expert models have more parameters per training/inference compute than dense models (Fedus et al., 2021; Du et al., 2021).”
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “As it stands, only large organizations have access to the latest generation of powerful language models (Brown et al., 2020; Rae et al., 2022; Fedus et al., 2021; Lieber et al., 2021; Tang, 2021).”