Investigating Contingency Awareness Using Atari 2600 Games

By Marc G. Bellemare, Joel Veness, Michael H. Bowling. 2012

101 citations recorded by the source corpus.

Contingency awareness is the recognition that some aspects of a future observation are under an agent's control while others are solely determined by the environment. This paper explores the idea of contingency awareness in reinforcement learning using the platform of Atari 2600 games. We introduce a technique for accurately identifying contingent regions and describe how to exploit this knowledge to generate improved features for value function approximation. We evaluate the performance of our techniques empirically, using 46 unseen, diverse, and challenging games for the Atari 2600 console. Our results suggest that contingency awareness is a generally useful concept for model-free reinforcement learning agents.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The Arcade Learning Environment: An Evaluation Platform for General Agents (2012, methodology): “ currently dicult to assess its general performance as the reported results were limited to only two games. Finally, the authors recently presented a domain independent feature generation technique (Bellemare et al., 2012) that attempts to focus its eort around the location of the player avatar. Thi”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, background): “[2] proposed a framework to learn contingent regions of an image affected by agent action, suggesting that contingency awareness is useful for model-free agents.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, background): “chemes to succeed. To date, the best agents require hundreds of millions of frames to attain nontrivial performance levels, and visit at best two or three rooms out of 72; most published agents (e.g. Bellemare et al., 2012; van Hasselt et al., 2016; Schaul et al., 2016; Mnih et al., 2016; Liang et a”
Progress & Compress: A scalable framework for continual learning (2018, positive): “Assessing P&C under more challenging conditions, we also consider the sequential learning of 6 games in the Atari suite (Bellemare et al., 2012) (“Space Invaders”, “Krull”, “Beamrider”, “Hero”, “Stargunner” and “Ms. Pac-man”).”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, methodology): “Using an action repeat of 4 is common in other domains like Atari (Bellemare et al., 2012; Mnih et al., 2015); however, we found that using an action repeat of 4 made the Hard-Eight tasks too difﬁcult for our demonstrators.”
Eigenoption Discovery through the Deep Successor Representation (2017, background): “The ability to recognize such features is known as contingency awareness, and it is known to have the potential to improve agents’ performance (Bellemare et al., 2012).”
Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning (2019, background): “The vast majority of evaluations follow a version of the approach described by Bellemare et al. (2012): ﬁrst researchers choose network architectures and tune hyperparameters on small set of Atari games; then they train agents using those hyperparameters on new games, reporting the learning curves…”
Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards (2019, background): “The idea of mega-reward originates from early psychology studies on contingency awareness [Watson, 1966; Baeyens et al., 1990; Bellemare et al., 2012], where infants are found to have awareness of how entities in their observation are potentially under their control.”
Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay (2016, background): “Approaches such as SARSA and contingency awareness [4] delivered promising results, but were far from human-level performance.”
A Neuroevolution Approach to General Atari Game Playing (2014, methodology): “ALE has been subsequently used as a domain for reinforcement learning and search agents [2], [3] and to investigate aspects of prediction and self-detection [4].”
Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games (2016, background): “There are a number of approaches to building ATARI game players, including some that do not use DL (e.g., Bellemare et al. 2012a, 2012b, 2013a, Lipovetzky et al. 2015).”
Human-level control through deep reinforcement learning (2015, methodology): “We compared DQN with the best performing methods from the reinforcement learning literature on the 49 games where results were available 12,15 .”