Measuring Intelligence through Games

By Tom Schaul, Julian Togelius, J. Urgen Schmidhuber. 2011

59 citations recorded by the source corpus.

Abstract Artiﬁcial general intelligence (AGI) refers to research aimed at tackling the full problemof artiﬁcial intelligence, that is, create truly intelligent agents. This sets it apart from mostAI research which aims at solving relatively narrow domains, such as character recognition,motion planning, or increasing player satisfaction in games. But how do we know when anagent is truly intelligent? A common point of reference in the AGI community is Legg andHutter’s formal deﬁnition of universal intelligence, which has the appeal of simplicity andgenerality but is unfortunately incomputable.Games of various kinds are commonly used as benchmarks for “narrow” AI research,as they are considered to have many important properties. We argue that many of theseproperties carry over to the testing of general intelligence as well. We then sketch how suchtesting could practically be carried out. The central part of this sketch is an extension ofuniversal intelligence to deal with ﬁnite time, and the use of sampling of the space of gamesexpressed in a suitably biased game description language.Keywords: measure of intelligence, games

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Evaluation & Benchmarks

Citations in the corpus

The Arcade Learning Environment: An Evaluation Platform for General Agents (2012, background): “Schaul, Togelius, and Schmidhuber (2011) also recently presented an interesting proposal for using games to measure the general capabilities of an agent.”
StarCraft II: A New Challenge for Reinforcement Learning (2017, neutral): “The ALE is a prominent example in a rich tradition of video game benchmarks for AI [30], including Super Mario [35], Ms Pac-Man [26], Doom [13], Unreal Tournament [11], as well as general video game-playing frameworks [29, 5] and competitions [23].”
Synthetic Data for Deep Learning (2019, background): “g proves to be helpful. 3.5 Computer games as virtual environments Computer games and game engines have been a very important source of problems and virtual environments for deep RL and AI in general [85, 296, 526]. Much of the foundational work in modern deep RL has been done in the en42 vironments”
General Video Game Playing (2013, background): “Recently, games were proposed as the specific ingredient for measuring this form of general intelligence (17); namely, an (infinite) set of unseen games were proposed to play the role of the “broad range of environments”.”
Imitation Learning (2017, methodology): “They can also be used as a testbed for artiﬁcial general intelligence [Schaul et al. 2011].”
A video game description language for model-based or interactive learning (2013, background): “Working backward from the long-term goal of using games to foster artificial general intelligence [2], a recent proposal has argued for extending the framework of general game playing (GGP) to video games [3], and to establish an AI competition where agents must demonstrate their proficiency…”
An Approximation of the Universal Intelligence Measure (2011, methodology): “Related discussion on the motivation behind such tests is given by Dowe and Hajek (1998); Hern´andez-Orallo (2000); Schaul et al. (2011).”
A Neuroevolution Approach to General Atari Game Playing (2014, background): “Broadly, this work is motivated by the idea of general-purpose intelligence [21], [25], applied to the domain of video game playing [37].”
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning (2019, background): “Arguably, targeting generalization is necessary in order to make progress on artiﬁcial general intelligence, rather than just solving individual problems (Schaul, Togelius, and Schmidhuber 2011).”
The 2014 General Video Game Playing Competition (2016, background): “In particular, it has been argued that an agent's intelligence can be estimated from its performance over a number of games sampled from a game space, and weighted by the lengths of their descriptions [14].”