HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

By Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning. 2018

4,936 citations recorded by the source corpus.

Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “We evaluate our model on the following tasks and datasets included in KILT: question answering: NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) and HotpotQA (Yang et al., 2018); slot ﬁlling: Zero Shot RE (Levy et al., 2017) and T-REx (Elsahar et al., 2018); entity linking:”
ReAct: Synergizing Reasoning and Acting in Language Models (2022, methodology): “We conduct empirical evaluations of ReAct and state-of-the-art baselines on four diverse benchmarks: question answering (HotPotQA, Yang et al., 2018), fact verification (Fever, Thorne et al., 2018), text-based game (ALFWorld, Shridhar et al., 2020b), and webpage navigation (WebShop, Yao et al.,…”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “For Round 1 contexts, we randomly sampled short multi-sentence passages from Wikipedia (of 250-600 characters) from the manually curated HotpotQA training set (Yang et al., 2018).”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “Yes/No questions make up a subset of the reading comprehension datasets CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), and Hot-PotQA (Yang et al., 2018), and are present in the ShARC (Saeidi et al., 2018) dataset.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “…challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from…”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, neutral): “We train on 140 GB of text drawing from a wide variety of domains: Wikipedia (En, De, Es, Fr), Project Gutenberg 1 , submissions from 45 subreddits, OpenWebText 2 , a large collection of news data (Hermann et al., 2015; Barrault et al., 2019; Sandhaus, 2008; Grusky et al., 2018), Amazon Reviews (McA”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “, 2017), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “Here we perform a study using self-consistency to see if it can help ﬁll in the gap, over a set of common NLP tasks, including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al., 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et”
Reasoning Like Program Executors (2022, background): “…on Reasoning Skills The literature focuses on reasoning skills, including numerical reasoning (Dua et al., 2019; Li et al., 2022a), multi-hop reasoning (Yang et al., 2018), reasoning in hybrid context (Chen et al., 2020b; Zhu et al., 2021) and logical reasoning (Liu et al., 2020; Yu et al., 2020).”
Big Bird: Transformers for Longer Sequences (2020, background): “HotpotQA-distractor [101]: Similar to natural questions, it requires ﬁnding the answer (Ans) as well as the supporting facts (Sup) over different documents needed for multi-hop reasoning from the given evidences.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “This paradigm has been taken even further by biasing the distribution of questions toward especially hard-to-model examples as in QAngaroo (Welbl et al., 2018), HotpotQA (Yang et al., 2018), and DROP (Dua et al., 2019).”
Longformer: The Long-Document Transformer (2020, background): “BPTransformer (Ye et al., 2019) evaluated on machine translation (MT), but didn’t explore the pretrainfinetune setting.”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, neutral): “In contrast to the rapid progress shown in Question Answering (QA) tasks (Rajpurkar et al., 2016; Joshi et al., 2017; Yang et al., 2018), the task of Question Generation (QG) remains understudied and challenging.”
ELI5: Long Form Question Answering (2019, background): “HotpotQA (Yang et al., 2018) extends this approach by building questions which challenge models to conduct multi-hop reasoning across multiple paragraphs, but the answer is still a short span.”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “For example, search ranking [1], open domain question answering [2], and fact veriﬁcation [3, 4] all ﬁrst retrieve relevant documents as the input to their later stage reranking, machine reading, and reasoning models.”
Break It Down: A Question Understanding Benchmark (2020, methodology): “When examining existing RC datasets (Yang et al., 2018; Dua et al., 2019), we observed that long spans in the question often match long spans in the text, due to existing practices of generating questions via crowdsourcing.”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “More details can be found in Appendix B. HotpotQA HotpotQA (Yang et al., 2018) is a human-annotated large-scale multi-hop QA dataset.”