Model-Based Reinforcement Learning for Atari

By Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milo's, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine and 4 more. 2019

995 citations recorded by the source corpus.

Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observations. However, this typically requires very large amounts of interaction -- substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a range of Atari games in low data regime of 100k interactions between the agent and the environment, which corresponds to two hours of real-time play. In most games SimPLe outperforms state-of-the-art model-free algorithms, in some games by over an order of magnitude.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “Generative modeling of video holds promise for applications such as content creation, forecasting, transfer learning and model-based reinforcement learning (Srivastava et al., 2015; Carl Vondrick, 2016; Oh et al., 2015; Kaiser et al., 2019).”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “Game Random Human SimPLe [20] Ape-X [18] R2D2 [21] MuZero MuZero normalized”
When to use parametric models in reinforcement learning? (2019, methodology): “The horizontal dashed line corresponds to the performance of SimPLe [Kaiser et al., 2019].”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “For example Ha & Schmid-huber (2018); Hafner et al. (2018); Kaiser et al. (2019) use pixel prediction losses to learn the latent representation while (Oh et al., 2017) chooses instead to optimize the model to predict next latent states with the same value function as the sampled next states.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “ural networks (Oh et al., 2015) have been used to provide accurate, long-term prediction of behavior in Atari games and have been combined with RL to improve sample-efﬁciency over model-free methods (Kaiser et al., 2019b). The visual foresight method involves training a video prediction model to pre”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “If the dynamics of the world are also learned, then it can be applied to planning [65, 39] or model-based RL [33, 50].”
Causally Correct Partial Models for Reinforcement Learning (2020, neutral): “…next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et al., 2018; Kaiser et al., 2019; Janner et al., 2019).”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “Following [19], we trained our Keypoint Q-Network (KeyQN) architecture for 100 , 000 interactions, which corresponds to 400 , 000 frames.”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Recent works have made progress in this regard (Finn et al., 2016; Ha and Schmidhuber, 2018; Kaiser et al., 2019; Kurutach et al., 2018; Sodhani et al., 2019; Zhang et al., 2018), but model-free alternatives are still hard to beat in terms of asymptotic performance.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “…2018), are appealing because they allow for algorithms that combine the sample efﬁciency of a model-based approach with the asymptotic performance of high-capacity function approximators, even in domains with high-dimensional observations (Oh et al., 2015; Ebert et al., 2018; Kaiser et al., 2019).”
Reinforcement Learning with Augmented Data (2020, background): “On the other hand, biological agents have the remarkable ability to learn quickly [10, 11], while being able to generalize to a wide variety of unseen tasks [12].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “, 2013) and significantly improve its data-efficiency on the Atari 100k (Kaiser et al., 2019) benchmark.”
Exploring Model-based Planning with Policy Networks (2019, methodology): “Latent dynamics models using VAE [19] are commonly used to solve problems with image input [12, 13, 15, 18].”
Dream to Control: Learning Behaviors by Latent Imagination (2019, methodology): “While agents that purely learn through world models are not yet competitive in these domains (Kaiser et al., 2019), the tasks offer a diverse test bed with visual complexity, sparse rewards, and early termination.”
Model-Based Reinforcement Learning with Value-Targeted Regression (2020, background): “…for building models, where the regression targets are often fixed objects (such as raw observations, features or keypoints; e.g. (Jaksch et al., 2010; Osband & Van Roy, 2014; Abbasi-Yadkori & Szepesvári, 2015; Xie et al., 2016; Agrawal & Jia, 2017; Yang & Wang, 2019a; Kaiser et al., 2019)).”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “…datasets has been studied in the context of model-based RL, which learns explicit forward models (Deisenroth & Rasmussen, 2011; Watter et al., 2015; Williams et al., 2015; Chua et al., 2018; Buckman et al., 2018; Wang & Ba, 2019; Kaiser et al., 2019; Hafner et al., 2019; Nagabandi et al., 2020).”