Multi-hop Question Answering via Reasoning Chains

By Jifan Chen, Shih-Ting Lin, Greg Durrett. 2019

88 citations recorded by the source corpus.

Multi-hop question answering requires models to gather information from different parts of a text to answer a question. Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process. We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer. We then feed the extracted chains to a BERT-based QA model to do final answer prediction. Critically, we do not rely on gold annotated chains or "supporting facts:" at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and coreference resolution. Nor do we rely on these annotations at test time, as our model learns to extract chains from raw text alone. We test our approach on two recently proposed large multi-hop question answering datasets: WikiHop and HotpotQA, and achieve state-of-art performance on WikiHop and strong performance on HotpotQA. Our analysis shows the properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way. Furthermore, human evaluation shows that our extracted chains allow humans to give answers with high confidence, indicating that these are a strong intermediate abstraction for this task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “More recently, the importance of diversity in the reasoning processes has been noticed, but only leveraged via task-specific training, either through an additional QA model over extracted reasoning paths (Chen et al., 2019), or by the introduction of latent variables in a commonsense knowledge graph”
Longformer: The Long-Document Transformer (2020, background): “Note that both approaches are significantly simpler than recent SOTA models that include pipeline approaches of multiple stages and customized architectures (e.g., (Tu et al., 2019; Chen et al., 2019; Tu et al., 2020; Groeneveld et al., 2020)).”
Rationale-Augmented Ensembles in Language Models (2022, methodology): “One relevant line of work tries to improve ratio-nalization and interpretability in natural language processing models, for example, by extracting rationales using task-speciﬁc approaches (Xu et al., 2021; Asai et al., 2020; Chen et al., 2019).”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, background): “, 2020) have targetted smaller subsets to develop models with interpretable reasoning (15)Similar approaches have been used to produce reasoning chains for multi-hop datasets (Chen et al., 2019; Kundu et al., 2019; Xiao et al., 2019) question relevant to the input context.”
Hierarchical Graph Network for Multi-hop Question Answering (2019, neutral): “For answer span extraction, a two-layer MLP on top of BiLSTMs is used to calculate the logits of every position being the beginning and end points of the ground-truth span : (Min et al., 2019b) 55.20 69.63 - - - - ChainEx (Chen et al., 2019) 61.20 74.11 - - - - Baseline Model (Yang et al., 2018) 45.”
Exploiting Reasoning Chains for Multi-hop Science Question Answering (2021, methodology): “…explore the effectiveness of KG by either automatically constructing the graph using named entity or semantic role labeling (Qiu et al., 2019; Bosselut et al., 2019; Fang et al., 2020; Chen et al., 2019) or resorting to existing KG (Saxena et al., 2020; Zhang et al., 2020; Yasunaga et al., 2021).”
From LSAT: The Progress and Challenges of Complex Reasoning (2021, background): “Constructing correct multi-hop reasoning chains (conversation flows) has been a key challenge for these tasks, and it is an essential research direction to effectively model multi-hop reasoning paths over multiple passages[40, 41].”
F1 Is Not Enough! Models and Evaluation towards User-Centered Explainable Question Answering (2020, background): “In contrast to the human evaluation from Chen et al. (2019), we evaluate explanations in the context of the model answer, ask participants to rate the predictions along multiple dimensions and collect responses from 40 instead of 3 subjects.”
Natural Language Deduction through Search over Statement Compositions (2022, background): “This condition differs from prior work on question answering with multi-hop reasoning (Yang et al., 2018; Chen et al., 2019) or models that build a proof structure but do not generate new statements (Saha et al., 2020, 2021).”
Rethinking Search: Making Experts out of Dilettantes (2021, background): “Other forms of inductive biases such as multi-hop reasoning [2, 13, 107] might also be useful.”
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2021, background): “Several prior work attempts to learn to ﬁnd evidence sentences in unsupervised manners in multi-hop QA (Chen et al., 2019; Yadav et al., 2019; Perez et al., 2020), whereas our work uses evidentiality to improve the generator components via multi-task training for diverse knowledge-intensive tasks,…”
Multi-Step Inference for Reasoning over Paragraphs (2020, methodology): “All the models share the same hyperparameters that are shown in Table 4.4 Metrics Though ROPES was released using both exact match (EM) and F1 as metrics, we only report EM here, as F1 has been shown to correlate poorly with human judgments on ROPES (Chen et al., 2019a).”