Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications

By Peifeng Yu, Mosharaf Chowdhury. 2019

95 citations recorded by the source corpus.

GPU computing is becoming increasingly more popular with the proliferation of deep learning (DL) applications. However, unlike traditional resources such as CPU or the network, modern GPUs do not natively support fine-grained sharing primitives. Consequently, implementing common policies such as time sharing and preemption are expensive. Worse, when a DL application cannot completely use a GPU's resources, the GPU cannot be efficiently shared between multiple applications, leading to GPU underutilization. We present Salus to enable two GPU sharing primitives: fast job switching and memory sharing, in order to achieve fine-grained GPU sharing among multiple DL applications. Salus implements an efficient, consolidated execution service that exposes the GPU to different DL applications, and enforces fine-grained sharing by performing iteration scheduling and addressing associated memory management issues. We show that these primitives can then be used to implement flexible sharing policies such as fairness, prioritization, and packing for various use cases. Our integration of Salus with TensorFlow and evaluation on popular DL jobs show that Salus can improve the average completion time of DL training jobs by $3.19\times$, GPU utilization for hyper-parameter tuning by $2.38\times$, and GPU utilization of DL inference applications by $42\times$ over not sharing the GPU and $7\times$ over NVIDIA MPS with small overhead.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Multi-tenancy for AI accelerators is gaining traction as an effective way to improve resource utilization, thereby amortizing the upfront embodied carbon footprint of customized system hardware for AI at the expense of potential operational carbon footprint increase [71], [72], [73], [74], [75].”
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping (2020, background): “Salus [56] aims to support finegrained GPU sharing among DNN tasks.”
PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications (2020, methodology): “For example, although NVIDIA Multiple Process Sharing (MPS) [6] and Salus [7] allow multiple processes to use the same GPU, they require all processes’ data (e.”
Wavelet: Efficient DNN Training with Tick-Tock Scheduling (2021, methodology): “To reduce GPU accessing interference caused by multiple processes (Yu & Chowdhury, 2020), we create two threads and two model replicas inside a job’s training process and attach each model replica to one thread.”
Zico: Efficient GPU Memory Sharing for Concurrent DNN Training (2021, methodology): “Salus [44] tries to remove the switching overhead by making model parameters of a job resident in GPU memory even when the job is inactive.”