Answering Complex Questions Using Open Information Extraction

By Tushar Khot, Ashish Sabharwal, Peter Clark. 2017

114 citations recorded by the source corpus.

While there has been substantial progress in factoid question-answering (QA), answering complex questions remains challenging, typically requiring both a large body of knowledge and inference techniques. Open Information Extraction (Open IE) provides a way to generate semi-structured knowledge for QA, but to date such knowledge has only been used to answer simple questions with retrieval-based methods. We overcome this limitation by presenting a method for reasoning with Open IE knowledge, allowing more complex questions to be handled. Using a recently proposed support graph optimization framework for QA, we develop a new inference model for Open IE, in particular one that can work effectively with multiple short facts, noise, and the relational structure of tuples. Our model significantly outperforms a state-of-the-art structured solver on complex questions of varying difficulty, while also removing the reliance on manually curated knowledge.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “TupleInference (Khot et al., 2017), also an ILP-based QA system, uses Open IE tuples (Banko et al., 2007) as its semi-structured representation.”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, background): “TupleInference (Khot et al., 2017), which performs semi-structured matching of the question with retrieved sentences, where the structure consists of Open IE tuples.”
Improving Question Answering with External Knowledge (2019, background): “Many researchers investigate how to directly or indirectly use automatically constructed knowledge bases/graphs from reference corpora (Khot et al., 2017; Kwon et al., 2018; Khashabi et al., 2018; Zhang et al., 2018) or existing external general knowledge graphs (Li and Clark, 2015; Sachan et al.”
QASC: A Dataset for Question Answering via Sentence Composition (2019, background): “Many systems answer science questions by composing multiple facts from semi-structured and unstructured knowledge sources (Khashabi et al. 2016; Khot, Sabharwal, and Clark 2017; Jansen et al. 2017; Khashabi et al. 2018b).”
SciTaiL: A Textual Entailment Dataset from Science Question Answering (2018, methodology): “Consequently, most systems attempt to stitch together multiple rules (Khot et al. 2015), table rows (Khashabi et al. 2016), or Open IE tuples (Khot, Sabharwal, and Clark 2017) to produce an answer.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “Traditionally, this idea has been used to compose outputs of lower-level language tasks to solve the higher-level NLU tasks (Harabagiu and Hickl, 2006; Lally et al., 2012; Khot et al., 2017; Khashabi et al., 2018b).”
WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference (2018, background): “…generally answering between 40% to 50% of multiple choice science questions correctly (Khot et al., 2015; Clark et al., 2016; Khashabi et al., 2016; Khot et al., 2017; Jansen et al., 2017, inter alia), and top-performing ensemble models nearly reaching a passing grade of 60% on middle school ( 8…”
Question Answering as Global Reasoning Over Semantic Abstractions (2018, methodology): “For the ﬁrst domain, we have a collection of question sets containing elementary-level science questions from standardized tests (Clark et al. 2016; Khot, Sabharwal, and Clark 2017).”
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project (2019, methodology): “Early work was dominated by information retrieval, statistical, and automated rule extraction and reasoning methods (Clark et al., 2014, 2016; Khashabi et al., 2016; Khot et al., 2017; Khashabi et al., 2018).”
Explainable Natural Language Reasoning via Conceptual Unification (2020, methodology): “Recent approaches have framed explanatory inference as the problem of building an optimal graph, whose generation is conditioned on a set of local and global semantic constraints (Khashabi et al., 2018; Khot et al., 2017; Jansen et al., 2017; Khashabi et al., 2016).”
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering (2019, methodology): “Unlike approaches that rely on structured KBs, which are expensive to create, (Khashabi et al., 2016; Khot et al., 2017; Zhang et al., 2018; Khashabi et al., 2018b; Cui et al., 2017; Bao et al., 2016), our method operates over KBs of only unstructured texts.”
Controlling Information Aggregation for Complex Question Answering (2018, methodology): “To mitigate this issue, recent QA methods that operate over OIE graphs limit themselves to reasoning with artiﬁcially short paths (Khot et al. 2017; Khashabi et al. 2017).”
KG^2: Learning to Reason Science Exam Questions with Contextual Knowledge Graph Embeddings (2018, methodology): “TupleInference: This model [Khot et al., 2017] searches for graph that best connects the terms in the question with an answer choice via the knowledge extracted by Open IE.”
Sanity Check: A Strong Alignment and Information Retrieval Baseline for Question Answering (2018, unknown): “nd Liu et al. [17] use explicit features alongside recurrent NN representations of questions and answers. ScienceQA: We compare against the most recently published works for this dataset. Khot et al. [13] employ Integer Linear Programming with a knowledge base of tuples to select the correct answers”
Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds (2018, methodology): “, 2004) and methods that build on extracted structured representations from both the question and the input text (Sachan et al., 2015; Sachan and Xing, 2016; Khot et al., 2017; Khashabi et al., 2018b).”
Multi-Modal Answer Validation for Knowledge-Based VQA (2021, background): “Such conversion is effective as statements occur much more frequently than questions in textual knowledge sources (Khot, Sabharwal, and Clark 2017).”
Case-Based Abductive Natural Language Inference (2020, methodology): “The results show that CB-ANLI (SentenceBERT) is third in the ranking, outperforming existing explainable systems based on Integer Linear Programming (ILP) (Khot et al., 2017; Khashabi et al., 2016) and pre-trained embeddings (Yadav et al., 2019a).”