A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

By Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner. 2021

547 citations recorded by the source corpus.

Readers of academic research papers often read with the goal of answering specific questions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information-seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present Qasper, a dataset of 5049 questions over 1585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers, motivating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, background): “Dasigi et al. (2021) study a subset of Qasper that has multiple annotated answers, and ﬁnd their overlap to be 60.9% F1, more than double our best baseline.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “, 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Kočiský et al.”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “5K NLP papers from ACL (Dasigi et al., 2021) 1M (1M) BREAKING NEWS† 20K latest articles from 400 English news sites (Baly et al.”
Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, methodology): “The LoCo Benchmark consists of 5 retrieval datasets, 3 from (Shaham et al., 2022) and 2 from (Dasigi et al., 2021).”
The NLP Task Effectiveness of Long-Range Transformers (2022, unknown): “(Chen et al., 2021; Huang et al., 2021), abstractive QA (aQA) (Dasigi et al., 2021; Pang et al., 2022).”
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, background): “L-Eval (An et al., 2023) and Long-Bench (Bai et al., 2023) are prominent examples, aggregating pre-existing tasks(Kociský et al., 2017; Dasigi et al., 2021; Yang et al., 2018; Huang et al., 2021; Joshi et al., 2017) 2023).”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, methodology): “Figure 1: An example from QuoRef (Dasigi et al., 2021) with the generated rationale shown in dark text.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “5K NLP papers from ACL (Dasigi et al., 2021) 1M (1M) BREAKING NEWS† 20K English news articles, scraped using (Ou-Yang, Lucas) 11M (1M) CONTRACTS† 500 commercial legal contracts (Hendrycks et al.”
Effective Long-Context Scaling of Foundation Models (2023, methodology): “, 2022) and Qasper (Dasigi et al., 2021), and 1-shot on QMSum (Zhong et al.”
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, unknown): “…is more targeted, evaluating models on specific types of long-context tasks, including document QA (Koˇcisk`y et al., 2018; Dua et al., 2019; Dasigi et al., 2021; Pang et al., 2022; Wang et al., 2024a), summarization (Zhong et al., 2021; Huang et al., 2021; Wang et al., 2022), retrieval and…”
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (2024, unknown): “The QASPER dataset includes 5,049 questions across 1,585 NLP papers, with each question probing for information embedded within the full text (Dasigi et al., 2021).”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “…from Longbench ( Bai et al., 2023), encompassing: narrative comprehension (Narrative QA (Koˇciský et al., 2018)), scientiﬁc understanding (QasperQA (Dasigi et al., 2021)), multi-hop reasoning (MultiField QA, HotpotQA (Yang et al., 2018), 2WikiMulti QA (Ho et al., 2020), Musique (Trivedi et al.,…”
SnapKV: LLM Knows What You are Looking for Before Generation (2024, methodology): “The experiments utilized a subset of the QASPER [37], with a fixed prompt instructing the LLM to summarize the paper.”
Long-context LLMs Struggle with Long In-context Learning (2024, unknown): “3. long-document question-answer or summarization over Qasper (Dasigi et al., 2021).”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, unknown): “This is computed on English (EN) tasks, Chinese (ZH) tasks, and all (All) tasks, code tasks are included in both languages. choose 13 of the English datasets, including Qasper, MultiFieldQA-en, HotpotQA, 2WikiMultihopQA, GovReport, Multi-news, TREC, TriviaQA, SAM-Sum, PassageCount, PassageRetrieval-”