Orca: A Distributed Serving System for Transformer-Based Generative Models

By Gyeong-In Yu, Joo Seong Jeong. 2022

785 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, methodology): “Orca [60] is a state-of-the-art LLM serving system optimized for throughput.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “Given the rapid development of LLM, there is an emergence of systems that are specialized for LLM inference, such as Faster Transformer (NVIDIA), Orca (Yu et al., 2022), LightSeq (Wang et al., 2021), PaLM inference (Pope et al., 2022), TurboTransformers (Fang et al., 2021), and Deepspeed-Inference (”
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving (2023, methodology): “(Pope et al., 2022) investigated the batching effect in scaling up LLM. Orca (Yu et al., 2022) proposed a distributed LLM serving system that uses a fine-grained scheduling policy to improve GPU utilization under various request lengths. vLLM (Kwon et al., 2023) used page tables to manage GPU…”
Splitwise: Efficient Generative LLM Inference Using Phase Splitting (2023, methodology): “Since vanilla vLLM only supports continuous batching with token preemption which can lead to much higher TBT, we implement state-of-the-art mixed continuous batching [81] as discussed earlier in Figure 2(c).”
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024, background): “Online Speculative (Liu et al., 2023) proposed an online KD strategy that dynamically aligns the drafter with the target LLM on the fly using the query data.”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “To achieve micro-batch pipelining, a new for-loop is added to Algorithm 1 to combine the iteration-level pipeline parallel execution schedule (Huang et al., 2019; Yu et al., 2022) with our single-device offloading runtime.”
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, methodology): “Among them, Orca [54] introduces continuous batching to increase through-put. vLLM [32] proposes paged-attention for fine-grained KV cache management.”
Yi: Open Foundation Models by 01.AI (2024, methodology): “We use dynamic batching [86] to minimize the response time and improve batching efficiency.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, methodology): “To address these limitations, we implemented sequence parallelism for distributed inference in VLMs. Unlike the training mode, the inference system must additionally manage tensors, such as input tokens and position encodings, that progressively change during the decoding phase (Yu et al., 2022).”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, methodology): “Thus, optimizing LLM inference has been a key focus for many recent systems [29,53,58,59,63,75,77].”
SGLang: Efficient Execution of Structured Language Model Programs (2023, neutral): “Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”