FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models
By Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, Qin Li, Qingwen Li. 2022
157 citations recorded by the source corpus.
The current trend in deep learning is to scale models to extremely large sizes with the objective of increasing their accuracy. Mixture-of-Expert (MoE) is the most popular pre-trained model that makes feasible the training of models with parameters beyond trillion-scale. Thanks to the dynamic activation of experts, i.e., shallow layers specialized in certain domains, it allows for sparse training of bigger models, removing the linearity between model size and computation. However, different from traditional deep learning models, it draws huge challenges to the efficiency of these training systems, including dynamic load imbalance, inefficient synchronous execution mode, and congested all-to-all communication. To address these challenges, we first propose a performance model that can both accurately predict the latency of different operations of a specific training task, and intuitively analyze its end-to-end performance via a novel roofline-like model. Then, guided by this model, we invent a dynamic shadowing approach to cope with load imbalance, and a smart fine-grained schedule that splits different operations and executes them concurrently. We design a congestion-avoiding expert selection strategy that relieves network congestion for the lower latency of iterations, when modification of expert selection is allowed. We implement and integrate the above optimizations as a general system, FasterMoE, empowering efficient distributed MoE model training. FasterMoE is evaluated on different cluster systems using up to 64 GPUs. It achieves 1.37X - 17.87X speedup compared with state-of-the-art systems for large models, including ZeRO, GShard, and BASE Layer. Source code of FasterMoE is now available at https://github.com/thu-pacman/FasterMoE.
Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Hardware and Architecture
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
In Computer Vision and Pattern Recognition
In Hardware and Architecture
Citations in the corpus
- MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022, neutral): “He et al. (2022) proposed FasterMoE, a system for distributed training of MoEs based on efficient communication strategies and changes to the MoE routing algorithm to avoid network congestion.”
- Tutel: Adaptive Mixture-of-Experts at Scale (2022, methodology): “Previous DL systems, including the latest MoE frameworks (Lepikhin et al., 2021; Ott et al., 2019; Rajbhandari et al., 2022; He et al., 2022), are mostly based on static runtime execution that does not fit dynamic MoE characteristics.”
- Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “Works such as FastMoE [45], DeepSpeed-MoE [118], Tutel [55], HetuMoE [108], FasterMoE [46] and MegaBlocks [32] focus on efficient distributed training and inference, including all-to-all communication scheduling, CUDA kernel optimizations, expert placement, and parallelism strategies.”