Adapting Language Models to Compress Contexts

By A. Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen. Conference on Empirical Methods in Natural Language Processing, 2023

364 citations recorded by the source corpus.

Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments and summary vectors from all previous segments are used in language modeling. We fine-tune OPT models on sequences of up to 30,720 tokens and show that AutoCompressors can utilize long contexts to improve perplexity. We evaluate AutoCompressors on in-context learning by compressing task demonstrations. We find that summary vectors are good substitutes for plain-text demonstrations, increasing accuracy while reducing inference cost. Finally, we explore the benefits of pre-computing summary vectors for large corpora by applying summary vectors to retrieval-augmented language modeling. Overall, AutoCompressors emerge as a simple and inexpensive solution for extending the context window of LMs while speeding up inference over long contexts.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/2f7364d8e5cf94315bf8905f57de9c5543e9a4bf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bill Yuchen Lin
2. Chao Du
3. Vineet Kosaraju
4. Qian Liu
5. Min Lin
6. Tianyu Pang
7. Jerry Tworek
8. Mo Bavarian

Citations in the corpus

• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, background): “An alternative approach is to compress or trim the context hidden or KV caches (Chevalier et al., 2023; Ge et al., 2024; Zhang et al., 2023; Liu et al., 2023; Xiao et al., 2024).”
• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition (2023, background): “It aligns with a broader research trend, where recent studies are actively exploring approaches to reduce the number of input tokens (Zhou et al., 2023; Ge et al., 2023; Chevalier et al., 2023; Jiang et al., 2023a; Li et al., 2023; Jiang et al., 2023b).”
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024, positive): “…techniques to condense lengthy histories into compact representations, whether in the form of LLM internal representations (Mu et al., 2023; Chevalier et al., 2023), discrete tokens (Jiang et al., 2023; Xu et al., 2024), or retrievable text segments via retrieval-augmented generation (RAG,…”
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, methodology): “(2) Soft prompt tuning methods like GIST (Mu et al., 2023), AutoCom-pressor (Chevalier et al., 2023), and ICAE (Ge et al., 2024), which require LLMs’ parameter fine-tuning, making them suitable for specific domains but not directly applicable to black-box LLMs. (3) Information-entropy-based…”