Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering

By Vikas Yadav, Steven Bethard, Mihai Surdeanu. 2019

107 citations recorded by the source corpus.

We propose an unsupervised strategy for the selection of justification sentences for multi-hop question answering (QA) that (a) maximizes the relevance of the selected sentences, (b) minimizes the overlap between the selected facts, and (c) maximizes the coverage of both question and answer. This unsupervised sentence selection can be coupled with any supervised QA model. We show that the sentences selected by our method improve the performance of a state-of-the-art supervised QA model on two multi-hop QA datasets: AI2’s Reasoning Challenge (ARC) and Multi-Sentence Reading Comprehension (MultiRC). We obtain new state-of-the-art performance on both datasets among systems that do not use external resources for training the QA system: 56.82% F1 on ARC (41.24% on Challenge and 64.49% on Easy) and 26.1% EM0 on MultiRC. Our justification sentences have higher quality than the justifications selected by a strong information retrieval baseline, e.g., by 5.4% F1 in MultiRC. We also show that our unsupervised selection of justification sentences is more stable across domains than a state-of-the-art supervised sentence selection method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Explainable Multi-hop Verbal Reasoning Through Internal Monologue (2021, methodology): “, 2019) have been successfully used in multi-hop reasoning problems (Banerjee et al., 2020; Asai et al., 2019; Yadav et al., 2019).”
Case-Based Abductive Natural Language Inference (2020, methodology): “…an optimal graph, conditioned on a set of semantic constraints (Khashabi et al., 2018; Khot et al., 2017; Jansen et al., 2017; Khashabi et al., 2016), or adopting iterative methods, using sparse or dense encoding mechanisms (Yadav et al., 2019a,b; Pirtoaca et al., 2019; Kundu et al., 2019).”
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2021, background): “Several prior work attempts to learn to ﬁnd evidence sentences in unsupervised manners in multi-hop QA (Chen et al., 2019; Yadav et al., 2019; Perez et al., 2020), whereas our work uses evidentiality to improve the generator components via multi-task training for diverse knowledge-intensive tasks,…”
Explainable Natural Language Reasoning via Conceptual Unification (2020, methodology): “In line with this work, we demonstrate that the use of uniﬁcation patterns for multi-hop explanations can enhance both accuracy and explainability of neural models on a challenging question answering task (Rajani et al., 2019; Yadav et al., 2019b).”
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (2024, methodology): “Both models were evaluated on a benchmark that consists of Winogrande [SBBC20], PIQA [BZB + 19], SciQ [WLG17], LAMBADA [PKL + 16], and ARC-easy [YBS19].”
Exploiting Reasoning Chains for Multi-hop Science Question Answering (2021, background): “For example,Yadav et al. (2019) extract evidence facts in consideration of their relevance, overlap and coverage.”
How Additional Knowledge can Improve Natural Language Commonsense Question Answering (2019, methodology): “Knowledge Retrieval: Systems for Information Retrieval, such as Elasticsearch (Gormley and Tong, 2015), has been used in prior work of (Khot et al., 2019; Pirtoaca et al., 2019; Yadav et al., 2019; Banerjee et al., 2019; Banerjee, 2019).”
Better & Faster Large Language Models via Multi-token Prediction (2024, methodology): “We evaluate the models from Section 3.7 on standard natural language processing benchmarks: ARC Challenge (Yadav et al., 2019), COPA (Roemmele et al., 2011), Hellaswag (Zellers et al., 2019), Natural Questions (Kwiatkowski et al., 2019), PIQA (Bisk et al., 2019), SIQA (Sap et al., 2019) and TriviaQA”