TensorFlow-Serving: Flexible, High-Performance ML Serving

By Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fang Li, Vinu Rajashekhar, Sukriti Ramesh, Jordan Soyke. 2017

362 citations recorded by the source corpus.

We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.

Subfields: Artificial Intelligence, Hardware and Architecture, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Efficiency & Compression

Citations in the corpus

Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, background): “Clipper [11], TensorFlow Serving [33], Nexus [45], InferLine [10], and Clockwork [20] are some earlier general model serving systems.”
IDK Cascades: Fast Deep Learning by Learning not to Overthink (2017, methodology): “The base models in the model cascades are treated as black-boxes and thus the proposed framework can be applied to any existing model serving systems [8, 9] with little modification.”
Factorized Deep Retrieval and Distributed TensorFlow Serving (2018, methodology): “…space, for addressing cold-start issues; 2) We developed a TensorFlow distributed WALS factorization trainer that can efficiently factorize a matrix with more than a billion rows and columns; 3) We designed and implemented a distributed serving system based on TensorFlow Serving [8].”
Low latency RNN inference with cellular batching (2018, methodology): “It is claimed that these operators “bear similarities to the batching approach of DyNet” [32].”
MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving (2019, methodology): “We choose three representative ML models, Inception-v3 [67], Inception-ResNet [66], and OpenNMT-ende [47], for common prediction tasks such as image classification and machine translation, and evaluate their peak inference performance with TensorFlow Serving [55].”
Nexus: a GPU cluster engine for accelerating DNN-based video analysis (2019, methodology): “The projects most closely related to Nexus are Clipper [6] and Tensorflow Serving [25].”
An anatomy for neural search engines (2019, methodology): “The final trained models were served with TensorFlow-Serving during the experiments [Olston et al., 2017].”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, methodology): “Accordingly, these systems focus on aspects such as batching the executions [7, 16, 35, 42, 56], selecting an appropriate model from multiple model variants [16,27,30,57], deploying multiple models (each for different inference services) on the same device [7, 29, 35, 56], and so on.”
Understanding Capacity-Driven Scale-Out Neural Recommendation Inference (2020, methodology): “Google’s data-center scale ML serving infrastructure, TensorFlow-Serving, encompasses model loading, versioning, RPC APIs, and inference batching [44].”
TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2018, background): “The ranking model builder handles this by generating a graph compatible with serving requirements and exporting it as a SavedModel [33]: a language agnostic object that can be loaded by the serving code, which can be in any low-level language such as C++ or Java.”
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up (2020, background): “In industry, TFS2 [42] is a proprietary model hosting service at Google, about which public information is not available.”