BookSum: A Collection of Datasets for Long-form Narrative Summarization

By Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, Dragomir R. Radev. 2021

225 citations recorded by the source corpus.

The majority of available text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such datasets will offer limited challenges for future generations of text summarization systems. We address these issues by introducing BookSum, a collection of datasets for long-form narrative summarization. Our dataset covers source documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of granularity of increasing difficulty: paragraph-, chapter-, and book-level. The domain and structure of our dataset poses a unique set of challenges for summarization systems, which include: processing very long documents, non-trivial causal and temporal dependencies, and rich discourse structures. To facilitate future work, we trained and evaluated multiple extractive and abstractive summarization models as baselines for our dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Recursively Summarizing Books with Human Feedback (2021, methodology): “Kry´sci´nski et al. (2021) report length being a confounder for BERTScore, with longer summaries having lower scores.”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, background): “BookSum (Kry´sci´nski et al., 2021) consists of public domain books and summaries of those books, chapters, and paragraphs.”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “Summarization of long documents has attracted significant interest in recent years (Cohan et al., 2018; Kryscinski et al., 2021).”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, result): “…in the dialogues, in contrast to existing summarization benchmarks of research papers (Li et al., 2023a), movie scripts (Chen et al., 2022), books (Kry´sci´nski et al., 2022), emails (Zhang et al., 2021b) etc. Traditional metrics like BLEU (Papineni et al., 2002) and ROGUE (Lin, 2004) focus on…”
Markov-Enhanced Clustering for Long Document Summarization: Tackling the 'Lost in the Middle' Challenge with Large Language Models (2025, unknown): “For our research, we initially explored multiple long-document summarization datasets, including SCROLLS [13] and FacetSum [14], but ultimately selected BookSum [12] as our primary dataset due to its superior document length characteristics.”
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents (2024, background): “BookSum is an English language narrative summarization dataset that provides human-written summaries at the level of paragraphs, chapters, and entire books (Kry´sci´nski et al., 2021).”
LLM as a Broken Telephone: Iterative Generation Distorts Information (2025, unknown): “In the BookSum dataset, the EN ↔ TH gradient is -0.026 (±0.014) with Llama and -0.040 (±0.025) with Mistral .”