LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

By Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu. Annual Meeting of the Association for Computational Linguistics, 2023

488 citations recorded by the source corpus.

In long context scenarios, large language models (LLMs) face three main challenges: higher computational cost, performance reduction, and position bias. Research indicates that LLM performance hinges on the density and position of key information in the input prompt. Inspired by these findings, we propose LongLLMLingua for prompt compression towards improving LLMs' perception of the key information to simultaneously address the three challenges. Our extensive evaluation across various long context scenarios demonstrates that LongLLMLingua not only enhances performance but also significantly reduces costs and latency. For instance, in the NaturalQuestions benchmark, LongLLMLingua boosts performance by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo, leading to substantial cost savings. It achieves a 94.0% cost reduction in the LooGLE benchmark. Moreover, when compressing prompts of about 10k tokens at ratios of 2x-6x, LongLLMLingua can accelerate end-to-end latency by 1.4x-2.6x. Our code is available at https://aka.ms/LongLLMLingua.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4c0428917aeee6aa7bd434f337d039f35996b736

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Boris Ginsburg
2. Chao Du
3. Vineet Kosaraju
4. Samuel Kriman
5. Qian Liu
6. Min Lin
7. Tianyu Pang
8. Jerry Tworek

Citations in the corpus

• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition (2023, background): “It aligns with a broader research trend, where recent studies are actively exploring approaches to reduce the number of input tokens (Zhou et al., 2023; Ge et al., 2023; Chevalier et al., 2023; Jiang et al., 2023a; Li et al., 2023; Jiang et al., 2023b).”
• Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “This approach not only captures the explicit information users seek but also un-covers deeper, transformative knowledge.”
• RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, background): “…2024a; Bulatov et al., 2023; Martins et al., 2022; Wu et al., 2022), retrieving relevant information from context (Xu et al., 2024a; Mohtashami & Jaggi, 2023; Wang et al., 2024; Tworkowski et al., 2024; Xiao et al., 2024a), or preserving the salient information via compression (Jiang et al., 2023).”
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, methodology): “For example, LongLLMLingua (Jiang et al., 2023b) applies a question-aware coarse-to-fine compression approach to estimate the information entropy of the tokens and adapts the estimation according to the question.”