SQuALITY: Building a Long-Document Summarization Dataset the Hard Way

By Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, Samuel R. Bowman. 2022

59 citations recorded by the source corpus.

Summarization datasets are often assembled either by scraping naturally occurring public-domain summaries—which are nearly always in difficult-to-work-with technical domains—or by using approximate heuristics to extract them from everyday text—which frequently yields unfaithful summaries. In this work, we turn to a slower but more straightforward approach to developing summarization benchmark data: We hire highly-qualified contractors to read stories and write original summaries from scratch. To amortize reading time, we collect five summaries per document, with the first giving an overview and the subsequent four addressing specific questions. We use this protocol to collect SQuALITY, a dataset of question-focused summaries built on the same public-domain short stories as the multiple-choice dataset QuALITY (Pang et al., 2021). Experiments with state-of-the-art summarization systems show that our dataset is challenging and that existing automatic evaluation metrics are weak indicators of quality.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Summarization

Citations in the corpus

GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “In this work we hire freelancers from the platform Upwork to write hard questions, as done in previous work on datasets for reading comprehension (Pang et al., 2022) and summarization (Wang et al., 2022).”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, background): “Our datasets present an almost even split be-tween automatically, crowdworker-, and expert-created datasets, with crowdworker-created ones being slightly more common, possibly confounded if experts were hired through crowdworking platforms, as was done for SQuality (Wang et al., 2022).”
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, unknown): “…et al., 2018; Dua et al., 2019; Dasigi et al., 2021; Pang et al., 2022; Wang et al., 2024a), summarization (Zhong et al., 2021; Huang et al., 2021; Wang et al., 2022), retrieval and attributing (Kamradt, 2023; Kuratov et al., 2024; Song et al., 2025; Laban et al., 2024; Zhang et al., 2024b;…”
RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (2025, unknown): “Building on the complementary properties of RAG and GraphRAG, we investigate the following two strategies to enhance overall QA performance.”