Efficient Attentions for Long Document Summarization

By Luyang Robby Huang, Shuyang Cao, Nikolaus Nova Parulian, Heng Ji, Lu Wang. 2021

442 citations recorded by the source corpus.

The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attentions. Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions. For evaluation, we present a new dataset, GovReport, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “GovReport (Huang et al., 2021): A summarization dataset of reports addressing various national policy issues published by the Congressional Research Service2 and the U.S. Government Accountability Office,3 where each document is paired with an expert-written executive summary.”
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, unknown): “Upon completing ∞ B ENCH , we assess the performance of several state-of-the-art (SOTA) long-context LLMs on this benchmark to gauge its difficulty and evaluate the effectiveness of these models.”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “…(MultiField QA, HotpotQA (Yang et al., 2018), 2WikiMulti QA (Ho et al., 2020), Musique (Trivedi et al., 2022)), document summarization (GovReport (Huang et al., 2021), QMSum (Zhong et al., 2021), MultiNews (Fabbri et al., 2019)), and various specialized tasks (TRec (Li & Roth, 2002), Trivia QA…”
SummScreen: A Dataset for Abstractive Screenplay Summarization (2021, background): “…articles (PubMed and arXiv; Cohan et al., 2018), multiple news articles (Multi-News; Fabbri et al., 2019), opinionated text (RottenTomatoes; Wang and Ling, 2016), Government reports (GovReport; Huang et al., 2021) and (extractive summarization of) chapters of novels (Ladhak et al., 2020).”
Extending Context Window of Large Language Models via Positional Interpolation (2023, methodology): “In particular, we consider the GovReport (Huang et al., 2021) dataset, which contains 17457 documents for training and 972 documents for evaluation.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Since the self-attention mechanism grows quadratically with sequence length, a number of studies aimed to improve the encoding efficiency of self-attention [79, 133].”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, methodology): “GovReport-QA (Cao and Wang, 2022) pair pre-existing summaries from GovReport (Huang et al., 2021) with annotator-written questions to form a hierarchical QFS task.”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “We extract GovReport from the original dataset (Huang et al., 2021).”
Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference (2024, unknown): “5-32k across a wide range of long-context datasets, including single-document QA: NarrativeQA, Qasper, Mul-tiFieldQA; multi-document QA: HotpotQA; summarization: GovReport; few-shot learning: TriviaQA.”
Retentive Network: A Successor to Transformer for Large Language Models (2023, background): “(Zhong et al., 2021), GovReport (Huang et al., 2021), SummScreen (Chen et al., 2021; Shaham et al., 2022”
YaRN: Efficient Context Window Extension of Large Language Models (2023, unknown): “To evaluate the long sequence language modeling performances, we use the GovReport [18] and Proof-pile [4] datasets both of which contain many long sequence samples.”
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, background): “It includes 17 datasets covering tasks such as single-document QA [20, 6], multi-document QA [33, 16], summarization [17, 35, 10], few-shot learning [21, 13, 19], synthetic, and code generation [14, 24].”
The NLP Task Effectiveness of Long-Range Transformers (2022, unknown): “(Chen et al., 2021; Huang et al., 2021), abstractive QA (aQA) (Dasigi et al., 2021; Pang et al., 2022).”
Efficient Streaming Language Models with Attention Sinks (2023, unknown): “(2020)), and summarization (GovReport (Huang et al., 2021), MultiNews (Fabbri et al., 2019)).”
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, unknown): “…QA (Koˇcisk`y et al., 2018; Dua et al., 2019; Dasigi et al., 2021; Pang et al., 2022; Wang et al., 2024a), summarization (Zhong et al., 2021; Huang et al., 2021; Wang et al., 2022), retrieval and attributing (Kamradt, 2023; Kuratov et al., 2024; Song et al., 2025; Laban et al., 2024; Zhang…”