Clipper: A Low-Latency Online Prediction Serving System

By Daniel Crankshaw, Xin Wang, Giulio Zhou, M. Franklin, Joseph E. Gonzalez, Ion Stoica. 2016

841 citations recorded by the source corpus.

Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learning frameworks and systems only address model training and not deployment. In this paper, we introduce Clipper, a general-purpose low-latency prediction serving system. Interposing between end-user applications and a wide range of machine learning frameworks, Clipper introduces a modular architecture to simplify model deployment across frameworks and applications. Furthermore, by introducing caching, batching, and adaptive model selection techniques, Clipper reduces prediction latency and improves prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. We evaluate Clipper on four common machine learning benchmark datasets and demonstrate its ability to meet the latency, accuracy, and throughput demands of online serving applications. Finally, we compare Clipper to the TensorFlow Serving system and demonstrate that we are able to achieve comparable throughput and latency while enabling model composition and online learning to improve accuracy and render more robust predictions.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

In ML Systems & Hardware

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “This increased cost is especially problematic in real-world applications, where the cost of inference dominates the cost of training (Jouppi et al., 2017; Crankshaw et al., 2017; Metz, 2017).”
Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “Training or inference over shared sub-models can beneﬁt from techniques that allow examples from different tasks to be combined in a single vectorized batch to get better accelerator utilization (Crankshaw et al., 2017).”
Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, background): “Clipper [11], TensorFlow Serving [33], Nexus [45], InferLine [10], and Clockwork [20] are some earlier general model serving systems.”
Ray: A Distributed Framework for Emerging AI Applications (2017, methodology): “, Horovod [53] for distributed training, Clipper [19] for serving, and CIEL [40] for simulation), in practice this approach is untenable due to the tight coupling of these components within applications.”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, methodology): “Model serving systems: Systems such as Clipper [37], TensorFlow-Serving [56], Clockwork [45] and Batch-Maker [44] study various placement, caching and batching strategies for model serving.”
Pond: CXL-Based Memory Pooling Systems for Cloud Platforms (2022, neutral): “After exporting to ONNX [49], the prototype adds the prediction (the size of zNUMA) on the VM request path using a custom inference serving system similar to [50, 51].”
Online Machine Learning in Big Data Streams (2018, background): “atch based on a precompiled sample, and then apply prediction for a live data stream. DSPE solutions are progressing in this area. One recent result, Clipper is a low-latency online prediction system [63] with a model abstraction layer that makes it easy to serve pre-trained models based on a large ”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, background): “Current general-purpose serving systems such as Triton Inference Server [7] and Clipper [16] serve as an abstraction for handling client requests and scheduling executions of the underlying execution engines.”
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up (2020, methodology): “Existing systems fundamentally assume that the constituent system components have unpredictable latency performance [16,58].”
InferLine: latency-aware provisioning and scaling for prediction serving pipelines (2020, methodology): “All experiments used Clipper [9] as the prediction-serving framework except for those in Fig.”
To Index or Not to Index: Optimizing Exact Maximum Inner Product Search (2017, methodology): “Prior systems, such as TUPAC [32], have shown that dense matrix multiply is an effective approach for model/parameter search, while other systems, such as Clipper [7], demonstrate that adaptive batching can lead to performance wins in online serving.”
MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving (2019, methodology): “Customers can then configure and deploy ML model serving softwares [24,30,68] on running instances to serve model inference requests.”
TensorFlow-Serving: Flexible, High-Performance ML Serving (2017, methodology): “Clipper and TensorFlow-Serving share a focus on remaining largely agnostic to the speciﬁc ML technology of the models being served, and have some similar components e.g. batching.”
Nexus: a GPU cluster engine for accelerating DNN-based video analysis (2019, methodology): “, drop a request only when it has already missed its deadline, and determine the batch size based on the time budget remaining for the earliest request in the queue (as in Clipper [6]).”
Making Contextual Decisions with Low Technical Debt (2016, methodology): “Clipper [15] incorporates feedback in real-time and thus better addresses (F3), but only explores over the predictions of existing (batch-trained) models.”
Rafiki: Machine Learning as an Analytics Service System (2018, methodology): “Also, Clipper optimizes the throughput, latency and accuracy separately, whereas Rafiki models them together to find the optimal model selection and batch size.”
TurboTransformers: an efficient GPU serving system for transformer models (2020, methodology): “serving optimization for variable-length input workloads. The request batching, which is the technique most helpful for performance, adopted in modern serving systems, such as TF-serving [9], Clipper [5], Nexus [26], are suitable for only fixed-length input. To solve the challenges of deploying an e”
Serverless Computing: One Step Forward, Two Steps Back (2018, methodology): “We have been working for some time on low-latency prediction serving in Clipper [5].”