WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference

By Peter Alexander Jansen, Elizabeth Wainwright, Steven Marmorstein, Clayton T. Morrison. 2018

135 citations recorded by the source corpus.

Developing methods of automated inference that are able to provide users with compelling human-readable justifications for why the answer to a question is correct is critical for domains such as science and medicine, where user trust and detecting costly errors are limiting factors to adoption. One of the central barriers to training question answering models on explainable inference tasks is the lack of gold explanations to serve as training data. In this paper we present a corpus of explanations for standardized science exams, a recent challenge task for question answering. We manually construct a corpus of detailed explanations for nearly all publicly available standardized elementary science question (approximately 1,680 3rd through 5th grade questions) and represent these as "explanation graphs" -- sets of lexically overlapping sentences that describe how to arrive at the correct answer to a question through a combination of domain and world knowledge. We also provide an explanation-centered tablestore, a collection of semi-structured tables that contain the knowledge to construct these elementary science explanations. Together, these two knowledge resources map out a substantial portion of the knowledge required for answering and explaining elementary science exams, and provide both structured and free-text training data for the explainable inference task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “F has three key characteristics of an ‘open book’: (a) it forms the basis for generating Q; (b) it has been deemed central to scientific explanations (Jansen et al., 2018); and (c) by itself, F is generally insufficient to answer questions in Q. Faced with a question q ∈ Q, a student or system S is…”
e-SNLI: Natural Language Inference with Natural Language Explanations (2018, background): “Jansen et al. [16] provided a dataset of natural language explanation graphs for elementary science questions.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, background): “WorldTree [13] 1,680 0 - - 3-5 natural 8 8 8 4 OpenBookQA [37] 5,957 0 10.”
QASC: A Dataset for Question Answering via Sentence Composition (2019, methodology): “We start with two medium size corpora of grade school level science facts: the WorldTree corpus (Jansen et al. 2018) and a collection of facts from the CK-12 Foundation.”
A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset (2018, background): “al., 2013; Jansen et al., 2016) concluded that advanced inference methods were necessary for many of the questions, as they could not be answered by simple fact-based retrieval. The Worldtree corpus (Jansen et al., 2018) augments more than 2,000 of the AI2 questions with explanation graphs (Jansen, ”
Multi-hop Question Answering via Reasoning Chains (2019, background): “Multi-hop question answering datasets explicitly require aggregating clues from different parts of some given documents (Dua et al., 2019; Welbl et al., 2018; Yang et al., 2018; Jansen et al., 2018; Khashabi et al., 2018).”
Explaining Answers with Entailment Trees (2021, methodology): “E NTAILMENT B ANK uses multiple-choice questions (and the correct answer option) from the ARC dataset of grade-school science questions (Clark et al., 2018), and a corpus of science-and general knowledge derived from WorldTree V2 (Xie et al., 2020; Jansen et al., 2018).”
Explanations for CommonsenseQA: New Dataset and Models (2021, background): “While existing work has looked at explaining the answer predicted by a model (Amini et al., 2019), we take up the task of explaining the given gold (correct) answer in a model oblivious fashion (Jansen et al., 2018).”
WorldTree V2: A Corpus of Science-Domain Structured Explanations and Inference Patterns supporting Multi-Hop Inference (2020, methodology): “…base contains a total of 66 tables whose relations and column structure were developed using a detailed data-driven analysis of the domain requirements (Jansen et al., 2016; Jansen et al., 2018), while several properties tables were drawn from the Aristo Tablestore (Khashabi et al., 2016).”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, background): “…et al., 2019; Narang et al., 2020; Brahman et al., 2021; Zhang et al., 2020), and (3) structured explanations consisting of explanations graphs (Jansen et al., 2018; Jansen and Ustalov, 2019; Xie et al., 2020; Saha et al., 2020; Kalyanpur et al., 2020; Saha et al., 2021), chain of facts (Khot…”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, background): “…2021) has explored different forms of explanations ranging from extractive ratio-nales (Zaidan et al., 2007; DeYoung et al., 2020), semi-structured, and structured explanations (Jansen et al., 2019; Mostafazadeh et al., 2020; Saha et al., 2021) to free-text explanations (Camburu et al., 2018).”
Case Study: Deontological Ethics in NLP (2020, methodology): “This has been used in scenarios where a knowledge base exists (Han et al., 2020; Jansen et al., 2018) for a QA system as well as in dynamic graph generation to reach the prediction (Liu et al., 2020; Rajagopal et al., 2020; Bosselut and Choi, 2019).”
On the Diversity and Limits of Human Explanations (2021, background): “Reference Task Questions and guidelines Jansen et al. (2018) science QA Specific interfaces were designed.”
Explainable Natural Language Reasoning via Conceptual Unification (2020, methodology): “The knowledge bases ( F KB and EKB ) are populated using the Worldtree corpus (Jansen et al., 2018) multiple-choice science questions.”
Autoregressive Reasoning over Chains of Facts with Transformers (2020, methodology): “The WorldTree dataset was designed speciﬁcally for ( > 2)-fact inference (Jansen et al., 2018; Xie et al., 2020): it consists of elementary science exam questions that can be explained by an average of 6 facts from a complementary dataset of textual facts.”
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning (2021, background): “…(3) structured explanations that range from semi-structured text (Ye et al., 2020) to chain of facts (Khot et al., 2020; Jhamtani and Clark, 2020; Gontier et al., 2020) to explanation graphs (based on edges between chains of facts) (Jansen et al., 2018; Jansen and Ustalov, 2019; Xie et al., 2020).”
Learning to Solve Complex Tasks by Talking to Agents (2021, background): “Note that such annotations are often very difﬁcult to obtain, especially given large knowledge bases (Jansen et al., 2018).”