My Computer Is an Honor Student - but How Intelligent Is It? Standardized Tests as a Measure of AI

By Peter Clark, Oren Etzioni. 2016

88 citations recorded by the source corpus.

Given the well-known limitations of the Turing Test, there is a need for objective tests to both focus attention on, and measure progress towards, the goals of AI. In this paper we argue that machine performance on standardized tests should be a key component of any new measure of AI, because attaining a high level of performance requires solving significant AI problems involving language understanding and world modeling - critical skills for any machine that lays claim to intelligence. In addition, standardized tests have all the basic requirements of a practical test: they are accessible, easily comprehensible, clearly measurable, and offer a graduated progression from simple tasks to those requiring deep understanding of the world. Here we propose this task as a challenge problem for the community, summarize our state-of-the-art results on math and science tests, and provide supporting datasets

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Datasets based on human standardized tests have also been used in AI for several years (e.g. Clark et al., 2016; Seo et al., 2014; Fujita et al., 2014; Strickland, 2013), and as part of the NTCIR QALab (NII, 2017), and for the 2016 Allen AI Science Challenge described in the Introduction (Schoenick…”
A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers (2020, methodology): “Human math/science tests have been considered more suitable for evaluating AI progress than the Turing test (Clark and Etzioni, 2016).”
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions (2016, background): “In this sense, it is a simple embodiment of an AI challenge requiring both language and reasoning, suitable as part of a broader Turing Test (Clark and Etzioni 2016), and with many potential applications if solved (e.g., tutoring systems, a scientist’s assistant).”
Relevance-guided Supervision for OpenQA with ColBERT (2020, background): “MRC refers to a family of tasks that involve answering questions about about text passages (Clark and Etzioni, 2016).”
Measuring abstract reasoning in neural networks (2018, background): “There is prior work emphasising both the advantages (Clark & Etzioni, 2016) and limitations (Davis, 2014) of apply-Measuring ing standardized tests in AI (see Marcus et al. (2016) and contributed articles for a review).”
Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems (2016, background): “Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 494–504, Valencia, Spain, April 3-7, 2017. c©2017 Association for Computational Linguistics”
A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset (2018, methodology): “While it is stated that SQuAD requires logical reasoning, the complexity of reasoning required is far less than that for the AI2 standardized tests dataset (Clark and Etzioni, 2016; Kembhavi et al., 2017).”
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project (2019, methodology): “For this reason, they have been used as a compelling—and challenging—task for the field for many years (Brachman et al., 2005; Clark and Etzioni, 2016).”
Controlling Information Aggregation for Complex Question Answering (2018, background): “The elementary science exam benchmark, for example, includes questions that test the student’s ability to reason over connected facts (Clark et al. 2013; Clark and Etzioni 2016; Jansen et al. 2016).”
Answering Science Exam Questions Using Query Rewriting with Background Knowledge (2018, result): “While it is stated that SQuAD requires logical reasoning, the complexity of reasoning required is far lesser than that required by the AI2 standardized tests dataset (Clark and Etzioni 2016; Kembhavi et al. 2017).”
What’s in an Explanation? Characterizing Knowledge and Inference Requirements for Elementary Science Exams (2016, background): “Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2956–2965, Osaka, Japan, December 11-17 2016.”
Survey on evaluation methods for dialogue systems (2019, methodology): “For multiple-choice RC the task is evaluated using accuracy (Clark and Etzioni 2016), that is, the number of times in which the system selected the correct answer.”
The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers (2018, background): “Hence, MWPs solvers are broadly considered as good test beds to evaluate the intelligence level of agents in terms of natural language understanding [4], [5] and the successful solving of MWPs would constitute a milestone towards general AI.”
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks (2020, background): “This capability would give rise to new opportunities for real-world applications such as education, medicine, and scientiﬁc discovery [ Clark and Etzioni, 2016 ] [ Clark, 2015 ] .”
Learning to Solve Geometry Problems from Natural Language Demonstrations in Textbooks (2017, methodology): “Solving Geometry Problems: Standardized tests have been recently proposed as ‘drivers for progress in AI’ (Clark and Etzioni, 2016 et al., 2014, 2015) and pre-university entrance exams (Fujita et al., 2014; Arai and Matsuzaki, 2014).”
Semantic Parsing to Probabilistic Programs for Situated Question Answering (2016, background): “Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 160–170, Austin, Texas, November 1-5, 2016. c©2016 Association for Computational Linguistics”