Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay

By Ionel-Alexandru Hosu, Traian Rebedea. 2016

107 citations recorded by the source corpus.

This paper introduces a novel method for learning how to play the most difficult Atari 2600 games from the Arcade Learning Environment using deep reinforcement learning. The proposed method, human checkpoint replay, consists in using checkpoints sampled from human gameplay as starting points for the learning process. This is meant to compensate for the difficulties of current exploration strategies, such as epsilon-greedy, to find successful control policies in games with sparse rewards. Like other deep reinforcement learning architectures, our model uses a convolutional neural network that receives only raw pixel inputs to estimate the state value function. We tested our method on Montezuma's Revenge and Private Eye, two of the most challenging games from the Atari platform. The results we obtained show a substantial improvement compared to previous learning approaches, as well as over a random player. We also propose a method for training deep reinforcement learning agents using human gameplay experience, which we call human experience replay.

Subfields: Artificial Intelligence, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Yoshua Bengio
4. Chelsea Finn
5. Quoc V. Le
6. Yuxi Li
7. Mohit Bansal
8. Percy Liang

In Sociology and Political Science

1. Marc Lanctot
2. Todd Hester
3. Peter Stone
4. Percy Liang
5. G. Tesauro
6. Joel Z. Leibo
7. Richard J. Chen
8. Bei Peng

Citations in the corpus

• Deep Q-learning From Demonstrations (2017, methodology): “Human Experience Replay (HER) (Hosu and Rebedea 2016) is an algorithm in which the agent samples from a replay buffer that is mixed between agent and demonstration data, similar to our approach.”
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration (2018, methodology): “…applying RL. Recent work extends this technique by (1) introducing different objective functions and regularization during pre-training, and (2) mixing demonstrations and rolled-out episodes during RL updates (Hosu & Rebedea, 2016; Hester et al., 2018; Vecerik et al., 2017; Nair et al., 2017).”
• Learning Montezuma's Revenge from a Single Demonstration (2018, methodology): “Starting episodes by resetting from demonstration states was previously proposed by Hosu and Rebedea [2016] for learning difficult Atari games.”
• One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL (2018, background): “This approach has been shown to be very beneficial to RL as well as imitation learning (Resnick et al., 2018; Hosu & Rebedea, 2016; Popov et al., 2017).”
• Hybrid Reinforcement Learning with Expert State Sequences (2019, background): “Hosu and Rebedea (2016) utilize expert state-action pairs to facilitate action value learning.”
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, methodology): “One example is the work done by Hosu and Rebedea (2016), in which they propose a deep RL method that combines online agent experiences with offline human experiences to train the agent more efficiently.”
• Backplay: "Man muss immer umkehren" (2018, unknown): “Hosu & Rebedea (2016) use uniformly random states of an expert demonstration as starting states for a policy.”