Compositional Questions Do Not Necessitate Multi-hop Reasoning

By Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke S. Zettlemoyer. 2019

174 citations recorded by the source corpus.

Multi-hop reading comprehension (RC) questions are challenging because they require reading and reasoning over multiple paragraphs. We argue that it can be difficult to construct large multi-hop RC datasets. For example, even highly compositional questions can be answered with a single hop if they target specific entity types, or the facts needed to answer them are redundant. Our analysis is centered on HotpotQA, where we show that single-hop reasoning can solve much more of the dataset than previously thought. We introduce a single-hop BERT-based RC model that achieves 67 F1—comparable to state-of-the-art multi-hop models. We also design an evaluation setting where humans are not shown all of the necessary paragraphs for the intended multi-hop reasoning but can still answer over 80% of questions. Together with detailed error analysis, these results suggest there should be an increasing focus on the role of evidence in multi-hop reasoning and possibly even a shift towards information retrieval style evaluations with large and diverse evidence collections.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Break It Down: A Question Understanding Benchmark (2020, methodology): “Similar to BREAKRC, retrieved paragraphs are fed to a pretrained BERT-based RC model (Min et al., 2019b) to answer x.”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “6 For Natural Questions Open, we use the English Wikipedia dump from December 20, 2018, following Lee et al. (2019) and Min et al. (2019a).”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, background): “(1) P AR R EADER reads each passage in parallel and predicts a candidate span from the passage and an answerable score (Min et al., 2019b; Wang et al., 2019).”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, methodology): “In this work, we use a modiﬁed version of BERT (Devlin et al., 2019) for multi-paragraph reading comprehension (Min et al., 2019).”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, background): “Unfortunately, the vast majority of existing benchmarks are based on crowdsourcing, which are by and large static and non-interactive, leading to datasets that are plagued with shortcuts and artifacts [18, 19, 20, 4].”
Flexible Generation of Natural Language Deductions (2021, background): “Following a line of work on multi-hop question answering (Welbl et al., 2018; Yang et al., 2018; Chen and Durrett, 2019; Min et al., 2019), the reading comprehension community has started to make inroads in the area of reasoning.”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “Unanticipated biases (Min et al., 2019; Manjunatha et al., 2019) are often introduced during dataset collection due to many reasons (eg., domain-speciﬁc contexts, crowd-workers distributions, etc.).”
Unsupervised Question Decomposition for Question Answering (2020, methodology): “D ECOMP RC (Min et al., 2019b) largely frames sub-questions as extractive spans of a multi-hop question, learning to predict span-based sub-questions via supervised learning on human annotations.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “Nevertheless, the presence of these gaps is well-documented (Gururangan et al., 2018; Poliak et al., 2018; Min et al., 2019), and Niven and Kao (2019) give an initial attempt at formally characterizing them.”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “However, there has been recent work (Jiang and Bansal, 2019; Min et al., 2019) showing that these datasets contain reasoning shortcuts and a large fraction of the questions can be answered with single-hop reasoning.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “On this resulting subset of 253 questions, MODULARQAis comparable to the DecompRC system with only a 2pt F1 gap.”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, background): “tic patterns, domains, example difficulties, reasoning types, and more (Yang et al., 2018; Gururangan et al., 2018; Geva et al., 2019; Min et al., 2019; Kiela et al., 2021).”
Evaluating NLP Models via Contrast Sets (2020, background): “Nevertheless, the presence of these gaps is well-documented (Gururangan et al., 2018; Poliak et al., 2018; Min et al., 2019; Kaushik and Lipton, 2018).”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
Hierarchical Graph Network for Multi-hop Question Answering (2019, methodology): “Then, a Machine Reading Comprehension (MRC) model is applied to the selected paragraphs for answer prediction (Nishida et al., 2019; Min et al., 2019b).”
Multi-hop Question Answering via Reasoning Chains (2019, background): “It also shows there exist other challenges to address in the question answering piece, complementary to the proposed model, like decomposing the question into different pieces (Min et al., 2019b) to further improve the multi-hop QA performance.”
Concealed Data Poisoning Attacks on NLP Models (2021, background): “Adversarially-crafted poison examples are also not the only type of low quality data; social (Sap et al., 2019) and annotator biases (Gururangan et al., 2018; Min et al., 2019) can be seen in a similar light.”
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering (2020, background): “As we only consider the questions whose answers can be extracted from p , we do not add any modiﬁ-cations to the model unlike some previous studies in HotpotQA (Min et al., 2019b; Ding et al., 2019).”
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020, methodology): “Min et al. (2019) clas-siﬁed the comparison questions in HotpotQA in three types: multi-hop, context-dependent multi-hop, and single-hop.”