Overview of Todai Robot Project and Evaluation Framework of its NLP-based Problem Solving

By Akira Fujita, Akihiro Kameda, Ai Kawazoe, Yusuke Miyao. 2014

37 citations recorded by the source corpus.

We introduce the organization of the Todai Robot Project and discuss its achievements. The Todai Robot Project task focuses on benchmarking NLP systems for problem solving. This task encourages NLP-based systems to solve real high-school examinations. We describe the details of the method to manage question resources and their correct answers, answering tools and participation by researchers in the task. We also analyse the answering accuracy of the developed systems by comparing the systems answers with answers given by human test-takers.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Datasets based on human standardized tests have also been used in AI for several years (e.g. Clark et al., 2016; Seo et al., 2014; Fujita et al., 2014; Strickland, 2013), and as part of the NTCIR QALab (NII, 2017), and for the 2016 Allen AI Science Challenge described in the Introduction (Schoenick…”
My Computer Is an Honor Student - but How Intelligent Is It? Standardized Tests as a Measure of AI (2016, background): “, (Bringsjord and Schimanski, 2003; JETAI 2011; Beyer et al., 2005; Fujita et al., 2014), as they appear to require significant advances in AI technology while also being accessible, measurable, understandable, and motivating.”
Elementary School Science and Math Tests as a Driver for AI: Take the Aristo Challenge! (2015, background): “Standardized tests have often been proposed a challenge problem for AI (e.g., Brachman, 2005; Fujita et al., 2014), as they appear to require significant advances in AI technology while also being accessible, measurable, understandable, and motivating.”
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project (2019, methodology): “…systems were developed for parts of the University of Tokyo Entrance Exam, including maths, physics, English, and history (Strickland, 2013; NII, 2013; Fujita et al., 2014), although in some cases questions were modiﬁed or annotated before being given to the systems (e.g., Matsuzaki et al., 2014).”
From LSAT: The Progress and Challenges of Complex Reasoning (2021, methodology): “For example, the annotated formulas representing physics, mathematics and biology knowledge are usually assumed to be provided to solve corresponding problems [7, 11], while history and geography questions are primarily solved by retrieving relevant information supplemented by shallow reasoning to m”
Learning to Solve Geometry Problems from Natural Language Demonstrations in Textbooks (2017, methodology): “Solving Geometry Problems: Standardized tests have been recently proposed as ‘drivers for progress in AI’ (Clark and Etzioni, 2016 et al., 2014, 2015) and pre-university entrance exams (Fujita et al., 2014; Arai and Matsuzaki, 2014).”
Easy Questions First? A Case Study on Curriculum Learning for Question Answering (2016, background): “Some example standardized tests are reading comprehensions (Richardson et al., 2013), algebra word problems (Kushman et al., 2014), geometry problems (Seo et al., 2014), entrance exams (Fujita et al., 2014; Arai and Mat-suzaki, 2014), etc.”
From Textbooks to Knowledge: A Case Study in Harvesting Axiomatic Knowledge from Textbooks to Solve Geometry Problems (2017, background): “…(Richardson et al., 2013; Sachan et al., 2015, inter alia), science question answering (Schoenick et al., 2016; Sachan et al., 2016, inter alia), algebra word problems (Kushman et al., 2014, inter alia), geometry problems (Seo et al., 2015), pre-university entrance exams (Fujita et al., 2014), etc.”