SGLang: Efficient Execution of Structured Language Model Programs

By Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez and 2 more. Neural Information Processing Systems, 2023

1,077 citations recorded by the source corpus.

Large language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are lacking for programming and executing these applications. We introduce SGLang, a system for efficient execution of complex language model programs. SGLang consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding. Experiments show that SGLang achieves up to 6.4x higher throughput compared to state-of-the-art inference systems on various large language and multi-modal models on tasks including agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat. The code is publicly available at https://github.com/sgl-project/sglang

Subfields: Artificial Intelligence, Software

View Paper: https://www.semanticscholar.org/paper/eb95c327260725498404eb43ec370d419b8d92c7

Citations in the corpus

• Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget (2026, unknown): “Supporting a new engine (e.g. SGLang (Zheng et al., 2024), TensorRT-LLM 4 ) is a new subclass—exactly how the vllm and hf backends already coexist behind the same contract: # <- add a subclass; client unchanged def score(self, req): ...”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “We omit detailed discussions here and refer readers to the original papers.”
• SHERLOC: Structured Diagnostic Localization for Code Repair Agents (2026, unknown): “To isolate the contribution of extended reasoning, we compare Qwen3-30B-A3B-Thinking against its non-thinking counterpart Qwen3-30B-A3B-Instruct , which shares the same parameter count, architecture, prompts, and tools.”