SCROLLS: Standardized CompaRison Over Long Language Sequences

By Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant and 1 more. 2022

184 citations recorded by the source corpus.

NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We examine existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input. SCROLLS contains summarization, question answering, and natural language inference tasks, covering multiple domains, including literature, science, business, and entertainment. Initial baselines, including Longformer Encoder-Decoder, indicate that there is ample room for improvement on SCROLLS. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We observe improvement on SCROLLS (Shaham et al., 2022), where the average input length is 3.5k, and no performance degradation on SQUAD (Rajpurkar et al., 2018).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “We use the Scrolls benchmark (Shaham et al., 2022) which comprises of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Kočiský et al., 2018), QuaLITY (Pang et al., 2021), and…”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “io/ quality/ and through the multitask SCROLLS benchmark (Shaham et al., 2022).”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, result): “This result is consistent with other work that report Longformer un-derperforms BART variants on the BookSum story summarization dataset (Pang et al., 2022) and the SCROLLS long-document generation benchmark (Shaham et al., 2022).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “As QA and NLI models may struggle with long inputs (Koˇciský et al., 2018; Pang et al., 2021; Yin et al., 2021; Shaham et al., 2022), metrics based on them may fail when handling long text.”
The NLP Task Effectiveness of Long-Range Transformers (2022, methodology): “Concurrent to our efforts, Shaham et al. (2022) propose a suite of text-to-text tasks as a long sequence NLP benchmark.”
Lost in the Middle: How Language Models Use Long Contexts (2023, background): “In contrast, the SCROLLS benchmark (Shaham et al., 2022) and its zero-shot extension ZeroSCROLLS (Shaham et al.”
A Length-Extrapolatable Transformer (2022, background): “, 2021) from SCROLLS (Shaham et al., 2022) with above 9k length on average.”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, background): “Concurrently, ZeroSCROLLS (Shaham et al., 2022, 2023) and L-Eval (An et al., 2023) are proposed as evaluation benchmarks for long text modeling.”
Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “The performance of LLMs on large-sized input prompts is likely to be a topic of great interest in the artificial intelligence community [226].”
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, unknown): “Scaling attention to longer context will unlock new capabilities (modeling and reasoning over multiple long documents [24, 43, 49] and ﬁles in large codebases [30, 47]), new modalities (high-resolution images [10], audio [23], video [25]), and new applications (user interaction with long history…”
Extending Context Window of Large Language Models via Positional Interpolation (2023, methodology): “We have also included results from two baselines in existing SCROLLS Leaderboard (Shaham et al., 2022; Ainslie et al., 2023).”
Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, methodology): “The LoCo Benchmark consists of 5 retrieval datasets, 3 from (Shaham et al., 2022) and 2 from (Dasigi et al., 2021).”
Retentive Network: A Successor to Transformer for Large Language Models (2023, background): “(Zhong et al., 2021), GovReport (Huang et al., 2021), SummScreen (Chen et al., 2021; Shaham et al., 2022”
Unifying Language Learning Paradigms (2022, methodology): “We use the Scrolls benchmark (Shaham et al., 2022) which comprises of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Ko ˇ cisk ´ y et al., 2018), QuaLITY (Pang et al., 2021), and…”
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (2024, unknown): “There are six datasets in total, where QMSum (QM), Qasper (QASP), QuALITY (QLTY) are token from SCROLLS (Shaham et al., 2022) and HotpotQA (HQA) MuSiQue (MSQ), MultiFieldQA-en (MFQA) are token from LongBench (Bai et al., 2023).”