Optimus: an efficient dynamic resource scheduler for deep learning clusters

By Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, Chuanxiong Guo. European Conference on Computer Systems, 2018

511 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/93a06eb066fe58ed7d036e46e4cee53483e16bb8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, methodology): “Parameter Server (PS) [34, 55] and AllReduce [7, 16, 42, 64] are two widely-adopted gradient synchronization strategies.”
A generic communication scheduler for distributed DNN training acceleration (2019, background): “A potential solution is to cooperatively schedule the communication within the shared cluster, and possibly integrate it with DNN cluster job scheduler [17, 29] or Coflow scheduler [13].”
A Learning-Based Incentive Mechanism for Federated Learning (2020, background): “Evidences have shown that the relationship between model accuracy and the amount of training data is nonlinear [12], [13].”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, background): “We believe that our work not only offers practical insights for those working on LLM training, but also paves the way for future research in this rapidly evolving field.”
Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications (2019, methodology): “We assume that the duration is known or can be estimated using existing techniques [41].”
3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning (2018, background): “show that in-datacenter distributed training can demand more bandwidth than datacenter networks and GPU interconnects currently offer, identifying transmitting state changes as a major bottleneck in distributed ML (Strom, 2015; Alistarh et al., 2017; Wen et al., 2017; Zhang et al., 2017; Lin et al.,”
Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (2019, background): “Optimus [37] leverages the convergence curve to predict job remaining time for dynamic resource scheduling and reduces average job completion time.”
Gandiva: Introspective Cluster Scheduling for Deep Learning (2018, background): “Optimus [39] derives a proper number of parameter-servers and workers for MxNetbased deep learning jobs, which complements Gandiva in GPU cluster scheduling.”