Understanding Dataset Design Choices for Multi-hop Reasoning

By Jifan Chen, Greg Durrett. 2019

116 citations recorded by the source corpus.

Learning multi-hop reasoning has been a key challenge for reading comprehension models, leading to the design of datasets that explicitly focus on it. Ideally, a model should not be able to perform well on a multi-hop question answering task without doing multi-hop reasoning. In this paper, we investigate two recently proposed datasets, WikiHop and HotpotQA. First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets. Furthermore, we find spurious correlations in the unmasked version of WikiHop, which make it easy to achieve high performance considering only the questions and answers. Finally, we investigate one key difference between these datasets, namely span-based vs. multiple-choice formulations of the QA task. Multiple-choice versions of both datasets can be easily gamed, and two models we examine only marginally exceed a baseline in this setting. Overall, while these datasets are useful testbeds, high-performing models may not be learning as much multi-hop reasoning as previously thought.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Flexible Generation of Natural Language Deductions (2021, background): “Following a line of work on multi-hop question answering (Welbl et al., 2018; Yang et al., 2018; Chen and Durrett, 2019; Min et al., 2019), the reading comprehension community has started to make inroads in the area of reasoning.”
Compositional Questions Do Not Necessitate Multi-hop Reasoning (2019, methodology): “We focus on H OTPOT QA, which consists of multi-hop questions written to require reasoning over two para-Figure Parallel research from Chen and Durrett (2019) presents similar ﬁndings on H OTPOT QA.”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “, 2018) can be solved by single-hop models (Chen and Durrett, 2019; Min et al., 2019).”
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data (2020, background): “Annotation De-biasing As has been suggested in previous papers (Kaushik and Lipton, 2018; Chen and Durrett, 2019; Clark et al., 2019), the existing benchmarks on multi-hop reasoning question answering have annotation biases, which makes designing multi-hop models unnecessary.”
Hierarchical Graph Network for Multi-hop Question Answering (2019, background): “For answer span extraction, a two-layer MLP on top of BiLSTMs is used to calculate the logits of every position being the beginning and end points of the ground-truth span : (Min et al., 2019b) 55.20 69.63 - - - - ChainEx (Chen et al., 2019) 61.20 74.11 - - - - Baseline Model (Yang et al., 2018) 45.”
Multi-hop Question Answering via Reasoning Chains (2019, background): “One reason is that correlations have been noted between the question and answer options (Chen and Durrett, 2019), so that the quality of the extracted evidence contributes less to the models’ downstream performance.”
Selective Question Answering under Domain Shift (2020, background): “On 65% of the underconﬁ-dence errors, the correct answer was the only span in the passage that could plausibly answer the question, suggesting that the model arrived at the answer due to artifacts in HotpotQA that facilitate guesswork (Chen and Durrett, 2019; Min et al., 2019).”
Answering Complex Open-domain Questions Through Iterative Query Generation (2019, background): “able contexts, and thus does not always test the model’s capability of multi-hop reasoning (Chen and Durrett, 2019; Min et al., 2019a).”
Can NLI Models Verify QA Systems' Predictions? (2021, background): “, 2019), ill-constructed datasets (Chen and Durrett, 2019), contrast sets (Gardner et al.”
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering (2019, result): “This indicates that: (a) knowledge aggregation is beneficial for these tasks; (b) ROCC can robustly handle non-trivial cases of aggregation with larger values of k; and (c) similar to other QA methods (Chen and Durrett, 2019), performance”
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020, background): “Further, data analyses (Chen and Durrett, 2019; Min et al., 2019) revealed that many examples in HotpotQA do not require multi-hop reasoning to solve. Recently, to evaluate the internal reasoning of the reading comprehension system, Inoue et al. (2020) proposed a new dataset R4C that requires system”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, background): “Our observation partly explains why a few earlier studies (Min et al., 2019a; Chen and Durrett, 2019), which only evaluated final performance, drew the conclusion that HOTPOTQA does not benefit much from multi-step reasoning or question decomposition.”
Look at the First Sentence: Position Bias in Question Answering (2020, background): “Several studies in QA also found that QA models do not leverages the full information in the given passage (Chen et al., 2016; Min et al., 2018; Chen and Durrett, 2019; Min et al., 2019).”
MoreHopQA: More Than Multi-hop Reasoning (2024, background): “In the short-cuts analyses, several previous works (Min et al., 2019a; Chen and Durrett, 2019; Jiang and Bansal, 2019) also raised the issues about the multi-hop reasoning abilities of the models and the shortcuts in existing datasets.”
Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning (2019, background): “Multiple previous works (Chen and Durrett, 2019; Min et al., 2019a) have shown that models performing strongly on HotpotQA are not necessarily capable of compositional reasoning.”
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval (2021, background): “However, it is limited to only two-hop questions, many of which may not require strong multi-hop reasoning capabilities (see Chen and Durrett [3]; Wang et al. [26]), softening the retrieval challenges discussed in §1.”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, methodology): “Dataset Method used Used by / Investigated by HotPotQA Partial Baselines (Min et al., 2019; Sugawara et al., 2020; Chen and Durrett, 2019b) Adversarial Evaluation (Jiang and Bansal, 2019) Data Improvements (Jiang and Bansal, 2019) Arch/Training Improvements (Jiang and Bansal, 2019) Manual Analyses…”
♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2021, background): “high scores (Min et al., 2019a; Chen and Durrett, 2019) without performing any reasoning that at least connects the intended facts together (Trivedi et al.”