Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

By Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, Fan Yang. 2019

471 citations recorded by the source corpus.

With widespread advances in machine learning, a number of large enterprises are beginning to incorporate machine learning models across a number of products. These models are typically trained on shared, multi-tenant GPU clusters. Similar to existing cluster computing workloads, scheduling frameworks aim to provide features like high efficiency, resource isolation, fair sharing across users, etc. However Deep Neural Network (DNN) based workloads, predominantly trained on GPUs, differ in two significant ways from traditional big data analytics workloads. First, from a cluster utilization perspective, GPUs represent a monolithic resource that cannot be shared at a fine granularity across users. Second, from a workload perspective, deep learning frameworks require gang scheduling reducing the flexibility of scheduling and making the jobs themselves inelastic to failures at runtime. In this paper we present a detailed workload characterization of a two-month long trace from a multi-tenant GPU cluster in a large enterprise. By correlating scheduler logs with logs from individual jobs, we study three distinct issues that affect cluster utilization for DNN training workloads on multi-tenant clusters: (1) the effect of gang scheduling and locality constraints on queuing, (2) the effect of locality on GPU utilization, and (3) failures during training. Based on our experience running a large-scale operation, we provide design guidelines pertaining to next-generation cluster schedulers for DNN training workloads.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

In Optimization & Training

Citations in the corpus

Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “On the other hand, with each new generation of accelerators, ML clusters are becoming increasingly heterogeneous (Jeon et al., 2019; Chaudhary et al., 2020; Weng et al., 2022).”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “thereby amortizing the upfront embodied carbon footprint of customized system hardware for AI at the expense of potential operational carbon footprint increase (Gschwind et al., 2017; Jeon et al., 2019; Yu & Chowdhury, 2019; Ghodrati et al., 2020; Kao & Krishna, 2021).”
Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications (2019, background): “A comparison of achieved makespan using FIFO (in TensorFlow) and Salus is shown in Figure 12.”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, background): “A public dataset on a DNN cluster [35] indicates that 50% of hosts have CPU utilization lower than 30%.”
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery (2020, background): “, 2004); 3–23 hours for a multi-tenant GPU clusters for DNN training at Microsoft (Jeon et al., 2019).”
PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications (2020, methodology): “We have analyzed a production GPU training trace from Microsoft [19, 20].”
Wavelet: Efficient DNN Training with Tick-Tock Scheduling (2021, background): “Distributed DNN training typically requires all the training tasks to be launched at the same time, i.e., gang-scheduled (Mahajan et al., 2020; Jeon et al., 2019).”
Datacenter-Scale Analysis and Optimization of GPU Machine Learning Workloads (2021, background): “In [2], the average observed GPU utilization in a multi-tenant environment running deep learning training was 52%, which demonstrates the magnitude of some of the challenges we mentioned.”
AntMan: Dynamic Scaling on GPU Clusters for Deep Learning (2020, background): “GPU cluster scheduling Today, DL training jobs in multitenant production clusters are managed by infrastructures such as Kubernetes or YARN [9,28], where jobs are allocated on dedicated GPUs, leading to common low utilization [27].”
Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads (2020, methodology): “Even organizations with private clusters have accumulated different accelerator types over time [34]; anecdotally, our research group has NVIDIA Titan V, Titan X, and P100 GPUs in its private cluster.”
Themis: Fair and Efficient GPU Cluster Scheduling for Machine Learning Workloads (2019, background): “GPU clusters are known to be heavily contented [19], and we find this also holds true in the subset of the trace of ML apps we consider (Figure 1).”
Zico: Efficient GPU Memory Sharing for Concurrent DNN Training (2021, background): “GPUs are the workhorse in server infrastructure and yet becoming highly contended resources at the same time [20, 43].”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “As each worker is typically equipped with multiple GPUs, this scale is sufﬁciently large to push the statistical limits of SGD [32,43,50,98].”