From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project

By Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon and 4 more. 2019

106 citations recorded by the source corpus.

AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy!, but the rich variety of standardized exams has remained a landmark challenge. Even as recently as 2016, the best AI system could achieve merely 59.3 percent on an 8th grade science exam. This article reports success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90 percent on the exam’s nondiagram, multiple choice (NDMC) questions. In addition, our Aristo system, building upon the success of recent language models, exceeded 83 percent on the corresponding Grade 12 Science Exam NDMC questions. The results, on unseen test questions, are robust across different test years and different variations of this kind of test. They demonstrate that modern natural language processing methods can result in mastery on this task. While not a full solution to general question-answering (the questions are limited to 8th grade multiple-choice science) it represents a significant milestone for the field.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “Exploiting commonality across machine learning tasks has a rich history studied under transfer learning (Caruana, 1997; Clark et al., 2019b).”
Measuring Massive Multitask Language Understanding (2020, background): “…topics like grade school subjects for which models can already achieve strong performance (Clark et al., 2018; Khot et al., 2019; Mihaylov et al., 2018; Clark et al., 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al., 2017; Richardson et al., 2013).”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “The Allen Institute for AI’s ARISTO is a complex, multi-part hybrid that has significantly outperformed other systems on eighth-grade science exams (Clark et al., 2019).”
Representing Numbers in NLP: a Survey and a Vision (2021, background): “Project Aristo (Clark et al., 2019) solves elementary and high school science problems, which often involve numeric reasoning.”
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering (2022, methodology): “We seed G REASE LM with RoBERTa-Large (Liu et al., 2019) for our experiments on CommonsenseQA, AristoRoBERTa (Clark et al., 2019) for our experiments on Open-bookQA, and SapBERT (Liu et al., 2021) for our experiments on MedQA-USMLE, demonstrating G REASE LM’s generality with respect to language mode”
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, background): “This choice is motivated by fact that the science domain is considered particularly challenging for QA (Clark et al., 2013; Clark, 2015; Clark et al., 2019), and existing science benchmarks are known to involve widespread use of such knowledge (see Clark et al. (2018); Boratko et al. (2018) for…”
Think about it! Improving defeasible reasoning by first modeling the question scenario. (2021, background): “…et al., 2019; Guu et al., 2020), or with knowledge generated from a separate source (Shwartz et al., 2020; Bosselut et al., 2019); and injected either as extra sentences fed directly to the model (Clark et al., 2020), via the loss function (Tandon et al., 2018), or via attention (Ma et al., 2019).”
Autoregressive Reasoning over Chains of Facts with Transformers (2020, background): “Research has shown that pre-trained transformers are able to infer knowledge from paragraphs of text, which is why they are more suited to handle this formulation of the task (Liu et al., 2019; Clark et al., 2019).”
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (2021, methodology): “Methods with AristoRoBERTa use the textual evidence by Clark et al. (2019) as an additional input to the QA context.”