Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning

By Woosuk Kwon, Gyeong-In Yu, Eunji Jeong, Byung-Gon Chun. 2020

92 citations recorded by the source corpus.

Deep learning (DL) frameworks take advantage of GPUs to improve the speed of DL inference and training. Ideally, DL frameworks should be able to fully utilize the computation power of GPUs such that the running time depends on the amount of computation assigned to GPUs. Yet, we observe that in scheduling GPU tasks, existing DL frameworks suffer from inefficiencies such as large scheduling overhead and unnecessary serial execution. To this end, we propose Nimble, a DL execution engine that runs GPU tasks in parallel with minimal scheduling overhead. Nimble introduces a novel technique called ahead-of-time (AoT) scheduling. Here, the scheduling procedure finishes before executing the GPU kernel, thereby removing most of the scheduling overhead during run time. Furthermore, Nimble automatically parallelizes the execution of GPU tasks by exploiting multiple GPU streams in a single GPU. Evaluation on a variety of neural networks shows that compared to PyTorch, Nimble speeds up inference and training by up to 22.34$\times$ and 3.61$\times$, respectively. Moreover, Nimble outperforms state-of-the-art inference systems, TensorRT and TVM, by up to 2.81$\times$ and 1.70$\times$, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Yonghui Wu
3. Ruoming Pang
4. Joseph Dean
5. Jinyu Li
6. Yu Zhang
7. Tie-Yan Liu
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Quoc V. Le
3. Dacheng Tao
4. Yi Yang
5. Andrew Zisserman
6. Jitendra Malik
7. Xiaogang Wang
8. Ming-Hsuan Yang

In Hardware and Architecture

1. Sanjay Ghemawat
2. Michael Isard
3. S. Hand
4. Joseph Dean

In Optimization & Training

1. Michael Isard
2. Aakanksha Chowdhery
3. Ruoming Pang
4. Ryan Sepassi
5. Yonghui Wu
6. Joseph Dean
7. Hyeontaek Lim

Citations in the corpus

• Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “Recent work shows that finer-grained sharing can improve resource efficiency further: virtualizing accelerators (Yu et al., 2020; Gupta et al., 2011; Vijaykumar et al., 2016) avoids dedicating a whole accelerator to a single user.”