Nexus: a GPU cluster engine for accelerating DNN-based video analysis

By Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, Ravi Sundaram. 2019

335 citations recorded by the source corpus.

We address the problem of serving Deep Neural Networks (DNNs) efficiently from a cluster of GPUs. In order to realize the promise of very low-cost processing made by accelerators such as GPUs, it is essential to run them at sustained high utilization. Doing so requires cluster-scale resource management that performs detailed scheduling of GPUs, reasoning about groups of DNN invocations that need to be co-scheduled, and moving from the conventional whole-DNN execution model to executing fragments of DNNs. Nexus is a fully implemented system that includes these innovations. In large-scale case studies on 16 GPUs, when required to stay within latency constraints at least 99% of the time, Nexus can process requests at rates 1.8-12.7X higher than state of the art systems can. A long-running multi-application deployment stays within 84% of optimal utilization and, on a 100-GPU cluster, violates latency SLOs on 0.27% of requests.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, background): “Clipper [11], TensorFlow Serving [33], Nexus [45], InferLine [10], and Clockwork [20] are some earlier general model serving systems.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, background): “Accordingly, these systems focus on aspects such as batching the executions [7, 16, 35, 42, 56], selecting an appropriate model from multiple model variants [16,27,30,57], deploying multiple models (each for different inference services) on the same device [7, 29, 35, 56], and so on.”
PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications (2020, methodology): “Moreover, the memory footprints of inference tasks are also increasing—the models are getting larger, and request batching is prevalently used to increase throughput [3].”
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up (2020, background): “Increasingly, modern ML applications are composed of pipelines or cascades of DNNs [27, 35, 36, 52].”
InferLine: latency-aware provisioning and scaling for prediction serving pipelines (2020, neutral): “Nexus [31] configures DNN inference pipelines for videostreaming applications.”
SHEPHERD: Serving DNNs in the Wild (2023, methodology): “We compare S HEPHERD against Clock-work [10] and Nexus [6].”