Constructing Datasets for Multi-hop Reading Comprehension Across Documents

By Johannes Welbl, Pontus Stenetorp, Sebastian Riedel. 2017

579 citations recorded by the source corpus.

Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or document. Enabling models to combine disjoint pieces of textual evidence would extend the scope of machine comprehension methods, but currently no resources exist to train and test this capability. We propose a novel task to encourage the development of models for text understanding across multiple documents and to investigate the limits of existing methods. In our task, a model learns to seek and combine evidence — effectively performing multihop, alias multi-step, inference. We devise a methodology to produce datasets for this task, given a collection of query-answer pairs and thematically linked documents. Two datasets from different domains are induced, and we identify potential pitfalls and devise circumvention strategies. We evaluate two previously proposed competitive models and find that one can integrate information across documents. However, both models struggle to select relevant information; and providing documents guaranteed to be relevant greatly improves their performance. While the models outperform several strong baselines, their best accuracy reaches 54.5% on an annotated test set, compared to human performance at 85.0%, leaving ample room for improvement.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “…has seen a great deal of recent work (Ra-jpurkar et al., 2016; Joshi et al., 2017), and there have been many recent attempts to construct QA datasets that require advanced reasoning abilities (Yang et al., 2018; Welbl et al., 2018; Mi-haylov et al., 2018; Zellers et al., 2018; Zhang et al., 2018).”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “…passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mi-haylov et al., 2018; Zhang et al., 2018), or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al., 2018; Khashabi et al., 2018).”
CoQA: A Conversational Question Answering Challenge (2018, background): “, 2018), and multi-fact reasoning (Welbl et al., 2018; Khashabi et al., 2018; Talmor and Berant, 2018).”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, background): “To focus on the more challenging problem of reasoning across sentences, new datasets have been proposed for multi-step RC. QAnga-roo (Welbl et al., 2018) have used a knowledge-base to identify entity pairs (s, o) with a known relation, r, which is also supported by a multi-hop path in a set of…”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Morerecently, Welbl et al. (2017b) created the WikiHop dataset, containing questions that appear to require more than one Wikipedia document to answer (“multihop ques-tions”).”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, background): “Developing models that successfully reason over multiple parts of their input has attracted substantial attention recently, leading to the creation of many multi-step reasoning Question Answering (QA) benchmarks (Welbl et al., 2018; Talmor and Berant, 2018; Khashabi et al., 2018; Yang et al., 2018; ”
Big Bird: Transformers for Longer Sequences (2020, methodology): “WikiHop [96]: Chose correct option from multiple-choice questions (MCQ), by aggregating information spread across multiple documents given in the evidences.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “This paradigm has been taken even further by biasing the distribution of questions toward especially hard-to-model examples as in QAngaroo (Welbl et al., 2018), HotpotQA (Yang et al., 2018), and DROP (Dua et al., 2019).”
Longformer: The Long-Document Transformer (2020, methodology): “We include global attention to question tokens and answer candidates for WikiHop and to question tokens for TriviaQA.”
Break It Down: A Question Understanding Benchmark (2020, background): “This includes reasoning over images (Antol et al., 2015; Johnson et al., 2017; Suhr et al., 2019; Hudson and Manning, 2019), paragraphs (Dua et al., 2019), documents (Welbl et al., 2018; Talmor and Berant, 2018; Yang et al., 2018), tables (Pasupat and Liang, 2015) and more.”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, background): “Moreover, most previous graph-based approaches evaluate on questions that are explicitly written to encourage reasoning based on a chain of entities, such as WIKIHOP (Welbl et al., 2017) or HOTPOTQA (Yang et al.”
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale (2019, background): “Machine Reading at Scale First proposed and formalized in Chen et al. (2017), MRS has gained popularity with increasing amount of work on both dataset collection (Joshi et al., 2017; Welbl et al., 2018) and MRS model developments (Wang et al., 2018; Clark and Gardner, 2017; Htut et al., 2018).”
oLMpics-On What Language Model Pre-training Captures (2019, background): “Questions that require multi-hop reasoning, such as “Who is the director of the movie about a WW2 pacific medic?”, have recently drawn attention (Yang et al., 2018b; Welbl et al., 2018; Talmor and Berant, 2018) as a challenging task for contemporary models.”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, methodology): “Recent datasets like QAngaroo (Welbl et al., 2018) and C OM - PLEX W EB Q UESTIONS (Talmor and Berant, 2018) explore different approaches of using pre-existing knowledge bases (KB) with pre-deﬁned logic rules to generate valid QA pairs, to test QA models’ capability of performing multi-hop…”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, methodology): “Jeopardy (Tunguz, 2019) and 2WikiMultiHop (Welbl et al., 2018) tasks measure the ability of LLMs to do reasoning in less structured settings, where the answer has to be generated in free form, and the level of available context varies between none (2WikiMultiHop, Jeopardy), to relevant and…”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “…the option of “Yes” or “No” answers (Dua et al., 2019; Kwiatkowski et al., 2019), questions requiring reasoning over multiple sentences or documents (Welbl et al., 2018; Yang et al., 2018), questions requiring rule interpretation or context awareness (Saeidi et al., 2018; Choi et al., 2018;…”
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, background): “Unlike most existing datasets and benchmarks that evaluate each of these RAG components in isolation, FRAMES offers a comprehensive test bed to gain a clear understanding of the overall quality of RAG systems (Lin et al., 2022; Yang et al., 2018b; Welbl et al., 2017).”
Neural Approaches to Conversational AI (2018, neutral): “, 2016), WikiHop (Welbl et al., 2017), DRCD (Shao et al.”
The Web as a Knowledge-Base for Answering Complex Questions (2018, background): “While many QA datasets have been developed recently (Yang et al., 2015; Rajpurkar et al., 2016; Hewlett et al., 2016; Nguyen et al., 2016; Onishi et al., 2016; Hill et al., 2015; Welbl et al., 2017), they lack a focus on the importance of question decomposition.”