State of the Art Control of Atari Games Using Shallow Reinforcement Learning

By Yitao Liang, Marlos C. Machado, Erik Talvitie, Michael H. Bowling. 2015

115 citations recorded by the source corpus.

The recently introduced Deep Q-Networks (DQN) algorithm has gained attention as one of the first successful combinations of deep neural networks and reinforcement learning. Its promise was demonstrated in the Arcade Learning Environment (ALE), a challenging framework composed of dozens of Atari 2600 games used to evaluate general competency in AI. It achieved dramatically better results than earlier approaches, showing that its ability to learn good representations is quite robust and general. This paper attempts to understand the principles that underlie DQN's impressive performance and to better contextualize its success. We systematically evaluate the importance of key representational biases encoded by DQN's network by proposing simple linear representations that make use of these concepts. Incorporating these characteristics, we obtain a computationally practical feature set that achieves competitive performance to DQN in the ALE. Besides offering insight into the strengths and weaknesses of DQN, we provide a generic representation for the ALE, significantly reducing the burden of learning a representation for each game. Moreover, we also provide a simple, reproducible benchmark for the sake of comparison to future work in the ALE.

Subfields: Artificial Intelligence, Developmental and Educational Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Developmental and Educational Psychology

In Reinforcement Learning

Citations in the corpus

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, methodology): “Therefore, agents must either use generic encodings capable of representing all games (e.g., Liang et al., 2016), or be able to automatically learn representations.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, background): “in nontrivial performance levels, and visit at best two or three rooms out of 72; most published agents (e.g. Bellemare et al., 2012; van Hasselt et al., 2016; Schaul et al., 2016; Mnih et al., 2016; Liang et al., 2016) fail to match even a human beginner’s performance. We deﬁned an exploration bonu”
Target-driven visual navigation in indoor scenes using deep reinforcement learning (2016, methodology): “[31] use RL for making decisions in ATARI games.”
IQA: Visual Question Answering in Interactive Environments (2017, background): “Recent works employing deep architectures train large networks that can learn policies to directly map pixels to actions [46, 54, 55, 73, 74].”
Generalization and Exploration via Randomized Value Functions (2014, background): “(8)Interestingly, recent work has been able to reproduce similar performance using linear value functions (Liang et al., 2015).”
Learning values across many orders of magnitude (2016, methodology): “This is why approximations are regularly proposed, typically trading off accuracy to computation [Martens and Grosse, 2015], and sometimes focusing on a certain aspect such as input normalization [Desjardins et al., 2015, Ioffe and Szegedy, 2015].”
Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning (2016, methodology): “We refer the reader to Liang et al. (2016) for further discussion about DQN evaluation methods on the ALE benchmark.”
Deep Exploration via Randomized Value Functions (2017, background): “Though algorithms of this genre have achieved impressive outcomes, notably in games such as backgammon[17], Atari arcade games [18], and go [19], they use naive exploration schemes that can be highly inefficient.”
The Benchmark Lottery (2021, background): “While in some publications episodes terminate when the game is over [Bellemare et al., 2013, Hausknecht et al., 2014, Liang et al., 2015, Lipovetzky et al., 2015, Martin et al., 2017], while others papers choose to terminate the training episodes for a subset of the games when the agent loses a…”
Deep Reinforcement Learning: An Overview (2017, methodology): “…uses raw inputs without manual feature engineering to generate outputs, e.g., AlexNet (Krizhevsky et al., 2012) with raw pixels for image classiﬁcation, Seq2Seq (Sutskever et al., 2014) with raw sentences for machine translation, and DQN (Mnih et al., 2015) with raw pixels and score to play games.”
Value-Aware Loss Function for Model-based Reinforcement Learning (2017, background): “The features φ might either be manually-designed (e.g., [Silver et al., 2007; Liang et al., 2016]) or automatically generated data-dependently, e.g., [Petrik, 2007; Mahadevan and Maggioni, 2007; Parr et al., 2007; Mahadevan and Liu, 2010; Geramifard et al., 2011; Farahmand and Precup, 2012; B¨ohmer…”
Model-free, Model-based, and General Intelligence (2018, background): “In the first case, the features fi are associated with each of the 256 values of the 128 memory bytes; in the second case, the features are the B-PROST pixel features [Liang et al., 2016] defined and motivated by the neural net architecture underlying DQN [Mnih et al., 2015].”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “However, prior work has found it challenging to uphold this promise [7, 79, 118, 125], prompting many prior methods to employ separate objectives for representation learning and RL [31, 43, 70, 74, 89, 91, 100, 101, 117, 140, 143].”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, unknown): “…proto-value functions (Mahadevan and Maggioni, 2007), gradient-based learning (Yu and Bertsekas, 2009), and feature generation schemes such as tile coding (Sutton, 1996) and the domain-independent features used in some Atari 2600 game-playing agents (Bellemare et al., 2013; Liang et al., 2016).”
Count-Based Exploration in Feature Space for Reinforcement Learning (2017, methodology): “To implement LFA in the ALE we use the Blob-PROST feature set presented in [Liang et al., 2016].”