Learning to Compress Prompts with Gist Tokens

By Jesse Mu, Xiang Lisa Li, Noah D. Goodman. 2023

409 citations recorded by the source corpus.

Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of"gist"tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Prompting & Reasoning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Prompting & Reasoning

In Efficiency & Compression

Citations in the corpus

Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “[23] in simultaneous work propose adding special "gist" tokens which are trained to summarize the prompt so far, and find that the model is incapable of remembering specific details that should be copied into the output.”
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, background): “Recognizing this gap, researchers started examining the potential of pruning tokens within the KV cache of auto-regressive LLMs. Mu et al. (2023) learns to compress the prompts into a few special tokens to reduce memory pressure during caching.”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, methodology): “Most relevant to this work, Mu et al. (2024) optimized a token such that, when the tokens after it could not attend to the tokens before it (i.e., a context compression token), it would provide sufficient information to future tokens.”
Extending Context Window of Large Language Models via Positional Interpolation (2023, background): “Mu et al. (2023) suggested that this may prevent models from remembering speciﬁc details in the past inputs.”
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024, positive): “…compression, developing techniques to condense lengthy histories into compact representations, whether in the form of LLM internal representations (Mu et al., 2023; Chevalier et al., 2023), discrete tokens (Jiang et al., 2023; Xu et al., 2024), or retrievable text segments via retrieval-augmented…”
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, methodology): “Efficiency can be further enhanced by using advanced caching methods (Ge et al., 2023; Mu et al., 2023; Gim et al., 2023) and optimized attention modules during decoding (Dao et al., 2022; Dao, 2023).”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, positive): “Finally, some recent advances such as gisting tokens (Mu et al., 2023) can learn to compress the KV cache for documents, but their expensive eviction policies are difficult to deploy during generation.”
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, background): “Soft prompt tuning methods (Wingate et al., 2022; Mu et al., 2023) typically require fine-tuning for the specific task.”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, methodology): “(2) Soft prompt tuning methods like GIST (Mu et al., 2023), AutoCom-pressor (Chevalier et al., 2023), and ICAE (Ge et al., 2024), which require LLMs’ parameter fine-tuning, making them suitable for specific domains but not directly applicable to black-box LLMs. (3) Information-entropy-based…”
Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (2022, neutral): “We opt for replacing only the final token of an input prompt, which we verify experimentally, since compressing future generated text into tokens is an interesting and burgeoning direction in itself [32].”