Observe and Look Further: Achieving Consistent Performance on Atari

By Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David M. Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Vecerík and 3 more. 2018

127 citations recorded by the source corpus.

Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently. In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of $\gamma = 0.999$ (instead of $\gamma = 0.99$) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance of an average human on 40 games using a common set of hyper parameters. Furthermore, it is the first deep RL algorithm to solve the first level of Montezuma's Revenge.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “[30] Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Večerı́k, et al.”
Model-Based Reinforcement Learning for Atari (2019, background): “Human" from Table 3 in Pohlen et al. (2018).”
Sample Efficient Adaptive Text-to-Speech (2018, background): “Few-shot learning can also be leveraged in reinforcement learning, such as by imitating human Atari gameplay from a single recorded action sequence (Pohlen et al., 2018) or online video (Aytar et al., 2018).”
A Framework for Data-Driven Robotics (2019, background): “to the solution proposed in previous work [25], where fixed ratio batches are formed with agent and demonstration data.”
Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “Ape-X DQfD (Pohlen et al., 2018) uses an auxilliary temporal consistency (TC) loss to prevent the Q -values of target state-action pairs from changing; ablation analysis suggested that the TC loss was critical to performance.”
Exploration by Random Network Distillation (2018, methodology): “Signiﬁcant progress has been achieved by methods with access to either expert demonstrations (Pohlen et al., 2018; Aytar et al., 2018; Garmulewicz et al., 2018), special access to the underlying emulator state (Tang et al., 2017; Stanton & Clune, 2018), or both (Salimans & Chen, 2018).”
Regularized Behavior Value Estimation (2021, methodology): “Previously, hinge losses were used for regularization in RL, but with different goals from ours (Su et al., 2020; Pohlen et al., 2018).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “R2D2 also makes use of a transformed loss, introduced in (Pohlen et al., 2018), rather than using clipped rewards.”
Playing hard exploration games by watching YouTube (2018, background): “Imitation learning methods such as DQfD have yielded promising results for guiding agent exploration in sparse-reward tasks, both in game-playing [20, 32] and robotics domains [41].”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, methodology): “The demonstrations used in DQfD and Ape-X DQfD [26, 27] only range in score from 32,300 to 34,900.”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, background): “For example, Aytar et al. (2018), Pohlen et al. (2018) and Salimans and Chen (2018b) have shown that RL with demonstrations can address the hard exploration problem in Montezuma’s Revenge.”
A Survey of Deep Reinforcement Learning in Video Games (2019, background): “…[26] integrate six extensions to DQN CNN value-based, off-policy RUDDER [27] return decomposition CNN-LSTM value-based, off-policy Ape-X DQfD [28] transformed Bellman operator, temporal consistency loss CNN value-based, off-policy C51 [30] distributional Bellman optimality CNN value-based,…”
Reinforcement Learning with A* and a Deep Heuristic (2018, background): “…sensors per state experience replay can include a priority based on loss Schaul et al. (2015) (as implemented in accompanying code), exploration rewards based on curiosity can be added Burda et al. (2018), rewards can be clipped Pohlen et al. (2018), a target network can be used, and so on.”
Reward (Mis)design for Autonomous Driving (2021, background): “…discounted in the problem speciﬁcation —which is generally considered correct for episodic tasks [43, p. 68] like these—despite nearly all papers’ adherence to the current best practice of discounting future reward to aid deep reinforcement learning solutions (as discussed by Pohlen et al. [38]).”
Recurrent Experience Replay in Distributed Reinforcement Learning (2018, methodology): “Following the modified Ape-X version in Pohlen et al. (2018), we do not clip rewards, but instead use an invertible value function rescaling of the form h(x) = sign(x)( √ |x|+ 1 − 1) + x which results in the following n-step targets for the Q-value function: ŷt = h ( n−1∑ k=0 rt+kγ k + γnh−1 (…”