M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

By Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, J. Zhang, Yong Li, Wei Lin and 2 more. 2021

47 citations recorded by the source corpus.

Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 and Switch Transformer possessing hundreds of billions or even trillions of parameters. However, under limited resources, extreme-scale model training that requires enormous amounts of computes and memory footprint suffers from frustratingly low efficiency in model convergence. In this paper, we propose a simple training strategy called"Pseudo-to-Real"for high-memory-footprint-required large models. Pseudo-to-Real is compatible with large models with architecture of sequential layers. We demonstrate a practice of pretraining unprecedented 10-trillion-parameter model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days. Besides demonstrating the application of Pseudo-to-Real, we also provide a technique, Granular CPU offloading, to manage CPU memory for training large model and maintain high GPU utilities. Fast training of extreme-scale models on a decent amount of resources can bring much smaller carbon footprint and contribute to greener AI.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Large Language Models

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “MoE allows for extreme scaling in terms of model sizes, with recent work reaching 1.6, 1.75 and even 10 trillion [5, 15, 36] parameters.”
Designing Effective Sparse Expert Models (2022, methodology): “…in the span of the last twelve months, a milestone of efﬁciently training trillion parameter deep neural networks has been achieved by multiple groups (Fedus et al., 2021; Yang et al., 2021; Du et al., 2021), and most recently, Lin et al. (2021) introduced techniques to train a 10T parameter model.”
Tutel: Adaptive Mixture-of-Experts at Scale (2022, background): “It is leveraged by large-scale distributed DNN models by putting a cross-GPU layer that partially exchanges hidden features from different GPUs (Fedus et al., 2022; Lin et al., 2021; Riquelme et al., 2021).”
ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, methodology): “Yang et al. (2021) introduces the M6-T architecture and expert prototyping which splits experts into different groups and applies k top-1 routing procedures (contrasting with the top-k routing commonly used elsewhere).”
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (2022, methodology): “Following by this, more recent work [26, 27, 28] exhibit the sparsity advantage of MoE.”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “These models [20, 26, 27, 28, 29] use mixture-of-experts to select different parameters for each incoming example.”