Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

By Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, Baris Kasikci. Conference on Machine Learning and Systems, 2023

339 citations recorded by the source corpus.

The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM service providers. To efficiently use GPU resources and boost throughput, batching multiple requests has emerged as a popular paradigm; to further speed up batching, LLM quantization techniques reduce memory consumption and increase computing capacity. However, prevalent quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully leverage the capabilities of modern GPUs, such as 4-bit integer operators, resulting in sub-optimal performance. To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss. Atom significantly boosts serving throughput by using low-bit operators and considerably reduces memory consumption via low-bit quantization. It attains high accuracy by applying a novel mixed-precision and fine-grained quantization process. We evaluate Atom on 4-bit weight-activation quantization setups in the serving context. Atom improves end-to-end throughput by up to $7.73\times$ compared to the FP16 and by $2.53\times$ compared to INT8 quantization, while maintaining the same latency target.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9529e50807f36acf3d2e4af994b5803c47e4746a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kurt Keutzer
2. A. Gholami
3. Sehoon Kim
4. Coleman Hooper
5. Song Han
6. Xia Hu
7. Jiaming Tang
8. Michael W. Mahoney

In Computer Vision and Pattern Recognition

1. Kurt Keutzer
2. A. Gholami
3. Sehoon Kim
4. Song Han
5. Xia Hu
6. Coleman Hooper
7. Jiaming Tang
8. Michael W. Mahoney

Citations in the corpus

• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, background): “Unlike the well-studied weight quantization (Lin et al., 2023; Xiao et al., 2023a; Zhao et al., 2024), to the best of our knowledge, only a few studies applied the vanilla 4bit round-to-nearest quantization to KV cache (Sheng et al., 2023; Zhang et al., 2023; Zhao et al., 2024) due to the streaming…”
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, methodology): “† denotes the method used in ATOM [36].”
• Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference (2024, positive): “Note that this memory load reduction is universal across all models and is compatible with existing quantization mechanisms (Zhao et al., 2024).”
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, neutral): “Previous work relies on using a calibration set to characterize the outlier features and keeping them in higher precision for inference [Zhao et al., 2023, Ashkboos et al., 2023].”
• SpinQuant: LLM quantization with learned rotations (2024, neutral): “Prior research mitigates this challenge by trading quantization difficulty between weights and activations [46, 23] or employing mixed-precision to handle outliers [50].”