Learning and Querying Fast Generative Models for Reinforcement Learning

By Lars Buesing, Theophane Weber, Sébastien Racanière, S. M. Ali Eslami, Danilo Jimenez Rezende, David P. Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis and 1 more. 2018

135 citations recorded by the source corpus.

A key challenge in model-based reinforcement learning (RL) is to synthesize computationally efficient and accurate environment models. We show that carefully designed generative models that learn and operate on compact state representations, so-called state-space models, substantially reduce the computational costs for predicting outcomes of sequences of actions. Extensive experiments establish that state-space models accurately capture the dynamics of Atari games from the Arcade Learning Environment from raw pixels. The computational speed-up of state-space models while maintaining high accuracy makes their application in RL feasible: We demonstrate that agents which query these models for decision making outperform strong model-free baselines on the game MSPACMAN, demonstrating the potential of using learned environment models for planning.

Subfields: Artificial Intelligence, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Generative Models

Citations in the corpus

Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “However, prior work [4, 14, 20] remains far from the state of the art in visually rich domains, such as Atari 2600 games [2].”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “For example, Chua et al. (2018); Doerr et al. (2018); Hafner et al. (2018); Buesing et al. (2018); Feinberg et al. (2018); Buckman et al. (2018) train models to predict a reward and a distribution over next states, minimizing the negative log-probability of the true next state.”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “We design a latent dynamics model with both deterministic and stochastic components (Buesing et al., 2018; Chung et al., 2015).”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, neutral): “Graph-structured and stochastic RNNs. Buesing et al. (2018).”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “Learning effective latent dynamics models is becoming feasible through advances in deep learning and latent variable models (Krishnan et al., 2015; Karl et al., 2016; Doerr et al., 2018; Buesing et al., 2018).”
Learning Powerful Policies by Using Consistent Dynamics Model (2019, methodology): “We use the state-of-the-art Learning to Query model (Buesing et al., 2018) as our state space model.”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, methodology): “Other recent work has considered predictions that “jump” more than one step into the future, using either ﬁxed-offset jumps (Buesing et al., 2018) or heuristics such as predictability of the frame (Neitz et al., 2018; Jayaraman et al., 2019; Gregor et al., 2019) to choose which frames to predict.”
An investigation of model-free planning (2019, background): “, 2018), though there has been recent progress in that direction (Buesing et al., 2018; Ebert et al., 2018).”
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, background): “Buesing et al. (2018) compare several meth-ods of dynamic modeling and show that state-space models could learn good state representations that could be encoded and fed to the Imagination-Augmented Agent.”
Model-Based Stabilisation of Deep Reinforcement Learning (2018, background): “Atari), model-based learning has been less successful, despite plenty of model-learning approaches that demonstrably learn accurate environment models (Oh et al. 2015; Fu and Hsu 2016; Chiappa et al. 2017; Leibfried, Kushman, and Hofmann 2017; Wang, Kosson, and Mu 2017; Weber et al. 2017; Buesing et”
Consistent Generative Query Networks (2018, methodology): “Quantitative Comparisons: We quantitatively compare CGQN with sSSM (Buesing et al., 2018), SV2P (Babaeizadeh et al.”
Generative Neurosymbolic Machines (2020, methodology): “…the density of the possible worlds plays a crucial role, e.g., in world models required for planning and model-based reinforcement learning [22, 21, 1, 36, 24, 38, 23], most previous entity-based models can only synthesize artiﬁcial images by manually conﬁguring the representation, but not…”
KeyIn: Discovering Subgoal Structure with Keyframe-based Video Prediction (2019, methodology): “…method that plans directly towards the ﬁnal goal (No Subgoal), a method that stochastically predicts subgoals at a ﬁxed time offset (Jumpy, similar to Buesing et al. (2018)), a heuristic method that determines keyframes as points of peak surprise (Surprise), and a bottleneck-based subgoal…”
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model (2019, methodology): “Previous works have typically observed that mixed deterministic-stochastic state space models are more effective [16, 20, 4] than fully stochastic ones.”
Temporal Difference Variational Auto-Encoder (2018, methodology): “Several works (Chung et al., 2015; Lee et al., 2018; Archer et al., 2015; Fraccaro et al., 2016; Liu et al., 2017; Goyal et al., 2017; Buesing et al., 2018; Serban et al., 2017) fall into this framework.”
Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision (2020, unknown): “More similar to our work, modelbased reinforcement learning [5,10,7,18] planning in latent space typically assumes access to the low-dimensional states of the environment and plan the”