QuALITY: Question Answering with Long Input Texts, Yes!

By Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He and 2 more. 2021

240 citations recorded by the source corpus.

To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, background): “For QuALITY, we use the majority class (which is just above one quarter).”
Measuring Progress on Scalable Oversight for Large Language Models (2022, methodology): “To create a challenging task that requires model assistance, we ask human participants to answer QuALITY questions under a 5-minute time limit (roughly paralleling Pang et al., 2022; Parrish et al., 2022b,a).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “, 2018), QuaLITY (Pang et al., 2021), and ContractNLI (Koreeda & Manning, 2021).”
Teaching language models to support answers with verified quotes (2022, background): “…questions about the world, steadily improving (Cheng et al., 2021; Nakano et al., 2021; Roberts et al., 2020) on questionanswering benchmarks (Kwiatkowski et al., 2019; Lin et al., 2021; Pang et al., 2021), and serving generated samples to users via APIs (Brockman et al., 2020; Cohere, 2021).”
GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “Instead of a real-world expertise gap coming from years of professional training (as in our dataset), the expertise gap in QuALITY is simulated by familiarity with a long passage of text (which takes only up to 30m to read).”
The NLP Task Effectiveness of Long-Range Transformers (2022, unknown): “(Chen et al., 2021; Huang et al., 2021), abstractive QA (aQA) (Dasigi et al., 2021; Pang et al., 2022).”
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions (2022, methodology): “We create a dataset of answer explanations to long-context multiple choice questions from QuAL-ITY (Pang et al., 2021) as an initial step in this direction.”
Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions (2022, methodology): “We build on the existing passages, questions, and arguments from the dataset created by Parrish et al. (2022), which uses passages and questions from QuALITY (Pang et al., 2022).”
Unifying Language Learning Paradigms (2022, methodology): “We use the Scrolls benchmark (Shaham et al., 2022) which comprises of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Ko ˇ cisk ´ y et al., 2018), QuaLITY (Pang et al., 2021), and Contr”
Effective Long-Context Scaling of Foundation Models (2023, methodology): “, 2018), 2-shot on QuALITY (Pang et al., 2022) and Qasper (Dasigi et al.”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, methodology): “We assign stories to splits to be consistent with the QuALITY dataset (Pang et al., 2021b), so stories that appear in both datasets are assigned to the same split.”
The Claude 3 Model Family: Opus, Sonnet, Haiku (methodology): “…multilingual settings (MGSM) [26]; common-sense reasoning in HellaSwag [27], WinoGrande [28]; reasoning over text in DROP [29]; reading comprehension in RACE-H [30] and QuALITY [31] (see Table 6); coding in HumanEval [32], APPS [33], and MBPP [34]; and a variety of tasks in BIG-Bench-Hard [35, 36].”
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, unknown): “…evaluating models on specific types of long-context tasks, including document QA (Koˇcisk`y et al., 2018; Dua et al., 2019; Dasigi et al., 2021; Pang et al., 2022; Wang et al., 2024a), summarization (Zhong et al., 2021; Huang et al., 2021; Wang et al., 2022), retrieval and attributing…”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “As QA and NLI models may struggle with long inputs (Koˇciský et al., 2018; Pang et al., 2021; Yin et al., 2021; Shaham et al., 2022), metrics based on them may fail when handling long text.”