A Neuroevolution Approach to General Atari Game Playing

By Matthew J. Hausknecht, Joel Lehman, Risto Miikkulainen, Peter Stone. 2014

193 citations recorded by the source corpus.

This paper addresses the challenge of learning to play many different video games with little domain-specific knowledge. Specifically, it introduces a neuroevolution approach to general Atari 2600 game playing. Four neuroevolution algorithms were paired with three different state representations and evaluated on a set of 61 Atari games. The neuroevolution agents represent different points along the spectrum of algorithmic sophistication - including weight evolution on topologically fixed neural networks (conventional neuroevolution), covariance matrix adaptation evolution strategy (CMA-ES), neuroevolution of augmenting topologies (NEAT), and indirect network encoding (HyperNEAT). State representations include an object representation of the game screen, the raw pixels of the game screen, and seeded noise (a comparative baseline). Results indicate that direct-encoding methods work best on compact state representations while indirect-encoding methods (i.e., HyperNEAT) allow scaling to higher dimensional representations (i.e., the raw game screen). Previous approaches based on temporal-difference (TD) learning had trouble dealing with the large state spaces and sparse reward gradients often found in Atari games. Neuroevolution ameliorates these problems and evolved policies achieve state-of-the-art results, even surpassing human high scores on three games. These results suggest that neuroevolution is a promising approach to general video game playing (GVGP).

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Recurrent World Models Facilitate Policy Evolution (2018, neutral): “Evolution-based algorithms have even been able to solve difﬁcult RL tasks from high dimensional pixel inputs [1, 31, 45, 63].”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, methodology): “R tangular boxplots cor espond to memorizing-NEAT while hollow, pinched b xplots correspond to randomizedNEAT (Hausknecht et al., 2014).”
World Models (2018, methodology): “Evolution-based algorithms have even been able to solve difﬁcult RL tasks from high dimensional pixel inputs (Kout-nik et al., 2013; Hausknecht et al., 2013; Parker & Bryant, 2012).”
Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017, neutral): “For Atari, Hausknecht et al. [2014] obtain impressive results.”
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science (2017, positive): “While NEAT has shown some impressive empirical results 20 , in practice, NEAT and, most of its direct variants have dif ﬁ culty scaling due to their very large search space.”
Policy Optimization by Genetic Distillation (2017, methodology): “HyperNEAT (D’Ambrosio & Stanley, 2007), which extends on NEAT and uses CPPN-based indirect encoding, has been used to learn to play Atari games from raw game pixels (Hausknecht et al., 2014).”
Frame Skip Is a Powerful Parameter for Learning to Play Atari (2015, methodology): “Existing reinforcement learning (RL) approaches use static frame skip: HNEAT (Hausknecht et al. 2013) uses a frame skip of 0; DQN (Mnih et al.”
Distributed Deep Q-Learning (2015, background): “In particular, we discuss the neural network architecture, the iterative training algorithm, and a mechanism that improves training convergence stability.”
Neuromodulation Improves the Evolution of Forward Models (2016, methodology): “NE has been applied successfully for a variety of problems including Atari game playing [14], control [13, 30], and automatic feature selection [32].”
Discovering Reinforcement Learning Algorithms (2020, background): “Dashed lines correspond to state-of-the-art algorithms for each year [3, 13, 24, 23, 9, 14].”
Evolving simple programs for playing atari games (2018, methodology): “ which has not been well studied with CGP. Multiple neuroevolution approaches, including HyperNEAT, and CMA-ES were applied to pixel and object representations of the Atari games in Hausknecht et al. [8]. „e performance of the evolved object-based controllers demonstrated the di†culties of using raw”
Emergent Tangled Graph Representations for Atari Game Playing Agents (2017, methodology): “For example, both DQN and Hyper-NEAT employ neural network architectures consisting of > 800, 000 weights, all of which are computed for every decision (See Appendix A in [5] and ‘Model Architecture’ under Methods in [15]).”
The Benchmark Lottery (2021, background): “While in some publications episodes terminate when the game is over [Bellemare et al., 2013, Hausknecht et al., 2014, Liang et al., 2015, Lipovetzky et al., 2015, Martin et al., 2017], while others papers choose to terminate the training episodes for a subset of the games when the agent loses a…”
Self-Correcting Models for Model-Based Reinforcement Learning (2016, unknown): “For instance, Oh et al. (2015) learned models of Atari 2600 games, which are fully deterministic (Hausknecht et al. 2014); human players often perceive them as stochastic due to their complexity.”
The Impact of Determinism on Learning Atari 2600 Games (2015, methodology): “In order to test the efficacy of different methods for adding randomness, we re-use an agent from (Hausknecht et al. 2013), memorizing-NEAT, a memorizing agent evolved on fully-deterministic Atari games.”
Cooperation and communication in multiagent deep reinforcement learning (2016, methodology): “While ineffective at optimizing large networks, evolutionary methods could often discover compact networks that did a reasonable job at representing the value function or policy (Hausknecht et al., 2013).”