MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

By Zhen Zhang, Shuai Zheng, Yida Wang, George Karypis, Trishul M. Chilimbi, Mu Li, Xin Jin. 2022

57 citations recorded by the source corpus.

Existing general purpose frameworks for gigantic model training, i.e., dense models with billions of parameters, cannot scale efficiently on cloud environment with various networking conditions due to large communication overheads. In this paper, we propose MiCS, which Minimizes the Communication Scale to bring down communication overhead. Specifically, by decreasing the number of participants in a communication collective, MiCS can utilize heterogeneous network bandwidth, reduce network traffic over slower links, reduce the latency of communications for maintaining high network bandwidth utilization, and amortize expensive global gradient synchronization overhead. Our evaluation on AWS shows that the system throughput of MiCS is up to 2.89× that of the state-of-the-art large model training systems. MiCS achieves near-linear scaling efficiency, which is up to 1.27× that of DeepSpeed. MiCS allows us to train a proprietary model with 100 billion parameters on 512 GPUs with 99.4% weak-scaling efficiency, and it is able to saturate over 54.5% theoretical computation power of each GPU on a public cloud with less GPU memory and more restricted networks than DGX-A100 clusters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sebastian Gehrmann
2. Shijie Wu
3. Mark Dredze
4. Sam Shleifer
5. Myle Ott
6. Chien-chin Huang
7. Min Xu
8. David Stuart Rosenberg

In Computer Vision and Pattern Recognition

1. Sam Shleifer
2. Mark Dredze
3. Shijie Wu
4. Sebastian Gehrmann
5. Chien-chin Huang
6. Min Xu
7. Prabhanjan Kambadur
8. Ozan Irsoy

In Efficiency & Compression

1. Chien-chin Huang
2. Sebastian Gehrmann
3. Myle Ott
4. Sam Shleifer

Citations in the corpus

• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023, background): “MiCS [33] and FSDP differ in gradient communication strategies.”
• This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. (unknown): “MiCS [61] minimizes the communication scale on top of ZeRO for better scalability on the public cloud.”
• BloombergGPT: A Large Language Model for Finance (2023, background): “Similarly, during OPT training, Zhang et al. (2022a) noticed spikes in the gradient and activation norms, or divergences in the training perplexity.”