This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation.

By Lianmin Zheng, Zhuohan Li, UC Hao Zhang, UC Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Google Yanping Huang, Google Yanping Huang, Amazon Yida Wang, Amazon Yida Wang and 12 more

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

1. Joseph E. Gonzalez
2. Ion Stoica
3. Percy Liang
4. Ce Zhang
5. Tomas Pfister
6. Christopher R'e
7. Zhen Zhang
8. Mu Li

Citations in the corpus

• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023, methodology): “Further, compilerbased techniques such as Alpa [35], GSPMD [31], and FlexFlow [9] leverage profiling, performance modeling, user annotations and search to find the best configuration across the parallelism space of data, tensor and pipeline for a given cluster.”
• Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, background): “Therefore, it is necessary to partition them across distributed GPUs and execute them in a model parallel fashion [28, 63].”
• SHEPHERD: Serving DNNs in the Wild (2023, background): “It is possible, however, to break such large models into smaller partitions [36] to group them with other models for better multiplexing.”
• MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud (2022, background): “Alpa [72] and Unity [60] use hierarchical search to jointly optimize withinand between-device to look for a good parallel strategy.”
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “Serving a single 175 billion LLM requires at least 350GB GPU memory using specialized infrastructure (Zheng et al., 2022).”
• A Survey of Large Language Models (2023, unknown): “Currently, open-source libraries like DeepSpeed [64], Colossal-AI [118], and Alpa [180] can well support the three parallel training methods.”
• A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Furthermore, the synergy of expert parallelism [34], [105], [108], [163], [164] with other existing parallel strategies (tensor [165], [166], [167], pipeline [168], [169], [170], sequence parallelism [171], [172], [173]) has been investigated to enhance the scalability and efficiency of MoE models…”
• High-throughput Generative Inference of Large Language Models with a Single GPU (2023, background): “There are two kinds of model parallelisms: tensor and pipeline parallelism (Narayanan et al., 2021; Zheng et al., 2022).”
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, background): “DistServe is orthogonal to the large body of work on model parallelism in training [28,36,40,46,59].”