Elementary School Science and Math Tests as a Driver for AI: Take the Aristo Challenge!

By Peter Clark. 2015

97 citations recorded by the source corpus.

While there has been an explosion of impressive, data-driven AI applications in recent years, machines still largely lack a deeper understanding of the world to answer questions that go beyond information explicitly stated in text, and to explain and discuss those answers. To reach this next generation of AI applications, it is imperative to make faster progress in areas of knowledge, modeling, reasoning, and language. Standardized tests have often been proposed as a driver for such progress, with good reason: Many of the questions require sophisticated understanding of both language and the world, pushing the boundaries of AI, while other questions are easier, supporting incremental progress. In Project Aristo at the Allen Institute for AI, we are working on a specific version of this challenge, namely having the computer pass Elementary School Science and Math exams. Even at this level there is a rich variety of problems and question types, the most difficult requiring significant progress in AI. Here we propose this task as a challenge problem for the community, and are providing supporting datasets. Solutions to many of these problems would have a major impact on the field so we encourage you: Take the Aristo Challenge!

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

CoQA: A Conversational Question Answering Challenge (2018, background): “Our work parallels a growing interest in developing datasets that test specific reasoning abilities: algebraic reasoning (Clark, 2015), logical reasoning (Weston et al.”
Solving General Arithmetic Word Problems (2016, methodology): “Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”
Crowdsourcing Multiple Choice Science Questions (2017, background): “Science exam QA is a high-level NLP task which requires the mastery and integration of information extraction, reading comprehension and common sense reasoning (Clark et al., 2013; Clark, 2015).”
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions (2016, methodology): “Our goal is a system that performs well on Elementary Science tests, a grand challenge for AI because of the wide variety of knowledge and reasoning skills required (Clark 2015).”
Answering Complex Questions Using Open Information Extraction (2017, background): “Elementary-level science exams is one such domain, requiring complex reasoning (Clark, 2015).”
WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference (2018, background): “Standardized science exams have recently been proposed as a challenge task for question answering (Clark, 2015), as these questions have very challenging knowledge and inference requirements (Clark et al., 2013; Jansen et al., 2016), but are expressed in simple-enough language that the linguistic…”
A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset (2018, methodology): “The AI2 Science Questions dataset was introduced by Clark (2015) along with the Aristo Framework, which we build off of.”
Learning What is Essential in Questions (2017, neutral): “In domains with multi-sentence questions covering a wide array of subject areas, such as standardized tests for elementary level science, the challenge is even more pronounced (Clark, 2015).”
Science Question Answering using Instructional Materials (2016, methodology): “We propose an approach for answering multiplechoice elementary science tests (Clark, 2015) using the science curriculum of the student and other domain specific knowledge resources.”
Commonsense Knowledge Enhanced Embeddings for Solving Pronoun Disambiguation Problems in Winograd Schema Challenge (2016, background): “In recent years, many artiﬁcial intelligence (AI) challenges and competitions have been proposed for evaluating the intelligence of computers [1, 2, 3].”
Prerequisite Skills for Reading Comprehension: Multi-Perspective Analysis of MCTest Datasets and Systems (2017, methodology): “Therefore, we also had to consider difference between commonsense and domain-specific knowledge, such as is required in elementary science tests, say Clark, Harrison, and Balasubramanian (2013). We will therefore refine the categorization of commonsense according to discussions by LoBue and Yates (2”
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks (2020, background): “This capability would give rise to new opportunities for real-world applications such as education, medicine, and scientiﬁc discovery [ Clark and Etzioni, 2016 ] [ Clark, 2015 ] .”
Answering Science Exam Questions Using Query Rewriting with Background Knowledge (2018, methodology): “The AI2 Science Questions dataset was introduced by Clark (2015) along with the Aristo Framework, which we build off of.”
Combing Context and Commonsense Knowledge Through Neural Networks for Solving Winograd Schema Problems (2016, background): “In recent years, many AI challenges or competitions have been proposed to help evaluate the cognitive levels of machines (Levesque, Davis, and Morgenstern 2011; Weston et al. 2015; Clark 2015).”
From LSAT: The Progress and Challenges of Complex Reasoning (2021, unknown): “Math Tests which is for 6-11 year olds [81].”
Combining Natural Logic and Shallow Reasoning for Question Answering (2016, methodology): “We evaluate our entailment system on the Regents Science Exam portion of the Aristo dataset (Clark et al., 2013; Clark, 2015).”