Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
By Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D. Turney, Daniel Khashabi. 2016
148 citations recorded by the source corpus.
What capabilities are required for an AI system to pass standard 4th Grade Science Tests? Previous work has examined the use of Markov Logic Networks (MLNs) to represent the requisite background knowledge and interpret test questions, but did not improve upon an information retrieval (IR) baseline. In this paper, we describe an alternative approach that operates at three levels of representation and reasoning: information retrieval, corpus statistics, and simple inference over a semi-automatically constructed knowledge base, to achieve substantially improved results. We evaluate the methods on six years of unseen, unedited exam questions from the NY Regents Science Exam (using only non-diagram, multiple choice questions), and show that our overall system’s score is 71.3%, an improvement of 23.8% (absolute) over the MLN-based method described in previous work. We conclude with a detailed analysis, illustrating the complementary strengths of each method in the ensemble. Our datasets are being released to enable further research.
Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Ashish Sabharwal
- Yejin Choi
- Percy Liang
- Jianfeng Gao
- Peter Clark
In Retrieval-Augmented Generation
- Peter Clark
- Ashish Sabharwal
- Tushar Khot
- Daniel Khashabi
- Sebastian Riedel
- Luke S. Zettlemoyer
- Ming Zhou
- Christopher D. Manning
In Natural Language Processing
- Ashish Sabharwal
- Peter Clark
- Tushar Khot
- Mohit Bansal
- Hannaneh Hajishirzi
- Matt Gardner
- Jonathan Berant
- Luke S. Zettlemoyer
Citations in the corpus
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “also contain similar phenomena to what is in DROP (Koncel-Kedziorski et al., 2015; Kushman et al., 2014; Hosseini et al., 2014; Clark et al., 2016; Ling et al., 2017).”
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “Each answer choice is scored based on the average PMI across all pairs of question and answer n-grams.”
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Datasets based on human standardized tests have also been used in AI for several years (e.g. Clark et al., 2016; Seo et al., 2014; Fujita et al., 2014; Strickland, 2013), and as part of the NTCIR QALab (NII, 2017), and for the 2016 Allen AI Science Challenge described in the Introduction (Schoenick…”
- Improving Question Answering with External Knowledge (2019, methodology): “Preliminary experimental results on ARC and Open-BookQA datasets demonstrate the effectiveness of our proposed approaches.”
- Hypothesis Only Baselines in Natural Language Inference (2018, background): “Hypotheses are automatically paired with premise sentences from domain specific texts (Clark et al., 2016), and labeled ( ENTAILMENT , NEUTRAL ) by crowd-sourced workers.”
- Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, background): “e is the property p of the query; for MEDHOP there is only the single query type – interacts with. TF-IDF Retrieval-based models are known to be strong QA baselines if candidate answers are provided (Clark et al., 2016; Welbl et al., 2017). They search for individual documents based on keywords in t”
- QASC: A Dataset for Question Answering via Sentence Composition (2019, methodology): “The corpus of 73M web documents (281GB) originated from Clark et al. (2016).”
- SciTaiL: A Textual Entailment Dataset from Science Question Answering (2018, methodology): “Science QA task involves the challenging domain of schoollevel science exams, where questions often require complex reasoning to arrive at the correct answer (Clark et al. 2016).”
- Crowdsourcing Multiple Choice Science Questions (2017, methodology): “Aristo performance is slightly better on this set than on real science exams (where Aristo achieves 71.3% accuracy (Clark et al., 2016)).”
- DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension (2019, neutral): “Besides crowdsourcing, some datasets are collected from examinations designed by educational experts (Penas et al., 2014; Shibuki et al., 2014; Tseng et al., 2016; Clark et al., 2016; Lai et al., 2017), which aim to test human examinees.”
- Answering Complex Questions Using Open Information Extraction (2017, methodology): “(1) 4th Grade set (1220 train, 1304 test) is a 10x larger superset of the NY Regents questions (Clark et al., 2016), and includes professionally written licensed questions.”
- Improving Machine Reading Comprehension with General Reading Strategies (2018, methodology): “3% ) and especially large improvements on datasets ARC, Open-BookQA, and MCTest (Table 4).”
- WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference (2018, background): “…performance, with individual QA solvers generally answering between 40% to 50% of multiple choice science questions correctly (Khot et al., 2015; Clark et al., 2016; Khashabi et al., 2016; Khot et al., 2017; Jansen et al., 2017, inter alia), and top-performing ensemble models nearly reaching a…”
- Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “Additional examples include story completion (Schwartz et al., 2017) and multiple choice questions (Clark et al., 2016, 2018).”
- QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships (2018, methodology): “The first two are an information retrieval system and a word-association method, following the designs of Clark et al. (2016).”
- Learning What is Essential in Questions (2017, positive): “(C) We show that this classifier can be used to improve a surprisingly effective IR based QA system (Clark et al., 2016) by 4%-5% on previously used question sets and by 1.2% on a larger question set.”