Low latency RNN inference with cellular batching

By Pin Gao, Lingfan Yu, Yongwei Wu, Jinyang Li. 2018

103 citations recorded by the source corpus.

Performing inference on pre-trained neural network models must meet the requirement of low-latency, which is often at odds with achieving high throughput. Existing deep learning systems use batching to improve throughput, which do not perform well when serving Recurrent Neural Networks with dynamic dataflow graphs. We propose the technique of cellular batching, which improves both the latency and throughput of RNN inference. Unlike existing systems that batch a fixed set of dataflow graphs, cellular batching makes batching decisions at the granularity of an RNN "cell" (a subgraph with shared weights) and dynamically assembles a batched cell for execution as requests join and leave the system. We implemented our approach in a system called BatchMaker. Experiments show that BatchMaker achieves much lower latency and also higher throughput than existing systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, neutral): “To address this problem, fine-grained batching mechanisms, such as cellular batching [16] and iteration-level scheduling [60], have been proposed.”
Supporting Very Large Models using Automatic Dataflow Graph Partitioning (2018, background): “Furthermore, it is difficult to saturate a GPU with very small batch sizes [18, 19].”
TurboTransformers: an efficient GPU serving system for transformer models (2020, methodology): “To avoid the zero-padding problem in RNN, BatchMaker [8] breaks the computation graph into a graph of cellulars and dynamically decides the set of cellulars should be batched together.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, result): “Once a request arrives, BatchMaker breaks the dataflow graph for processing the request into RNN cells, schedules execution at the granularity of cells (instead of the entire graph), and batches the execution of identical cells (if any).”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, methodology): “Model serving systems: Systems such as Clipper [37], TensorFlow-Serving [56], Clockwork [45] and Batch-Maker [44] study various placement, caching and batching strategies for model serving.”