Recurrent Environment Simulators

By Silvia Chiappa, Sébastien Racanière, Daan Wierstra, Shakir Mohamed. 2017

225 citations recorded by the source corpus.

Models that can simulate how environments change in response to actions can be used by agents to plan and act efficiently. We improve on previous environment simulators from high-dimensional pixel observations by introducing recurrent neural networks that are able to make temporally and spatially coherent predictions for hundreds of time-steps into the future. We present an in-depth analysis of the factors affecting performance, providing the most extensive attempt to advance the understanding of the properties of these models. We address the issue of computationally inefficiency with a model that does not need to generate a high-dimensional image at each time-step. We show that our approach can be used to improve exploration and is adaptable to many diverse environments, namely 10 Atari games, a 3D car racing environment, and complex 3D mazes.

Subfields: Artificial Intelligence, Ocean Engineering, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Ocean Engineering

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, methodology): “As learned simulators of Atari environments are core ingredients of our approach, in many aspects our work is motivated by Oh et al. (2015) and Chiappa et al. (2017), however we focus on using video prediction in the context of learning how to play the game well and positively verify that learned…”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “…Igl et al., 2018; Srinivas et al., 2018), and pure video prediction without planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016; Chiappa et al., 2017; Babaeizadeh et al., 2017; Gemici et al., 2017; Denton and Fergus, 2018; Buesing et al., 2018; Doerr et al., 2018; Gregor and…”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “This weakness could be the reason that many previous works that learn dynamics models of RL environments do not actually use those models to fully replace the actual environments [8, 60].”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “Chiappa et al. (2017) successfully train deterministic state-space models for ALE modelling (largely equivalent to the considered dSSMs here); they however do not explore the computational complexity advantage of SSMs, and do not study RL applications of their models.”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, methodology): “Oh et al. (2015) and Chiappa et al. (2017) propose methods for learning observation-prediction models in the Atari domain, but apply these models only to improve exploration.”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “More recently, Chiappa et al. (2017) showed significant improvements over this original model, including the ability to plan with the internal state.”
Value Prediction Network (2017, background): “Another line of work [23, 4, 8, 30] uses observation-prediction models not for planning, but for improving exploration.”
The Predictron: End-To-End Learning and Planning (2016, background): “(Oh et al., 2015; Chiappa et al., 2016); but are yet to surpass state-of-the-art model-free methods in challenging RL domains with raw inputs (e.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…works have studied applying sequence models of various types to represent components in standard RL algorithms, such as policies, value functions, and models (Bakker, 2002; Heess et al., 2015a; Chiappa et al., 2017; Parisotto et al., 2020; Parisotto & Salakhutdinov, 2021; Kumar et al., 2020b).”
Toward Causal Representation Learning (2021, background): “Particularly relevant for causal leaning are generative world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see I NTRODUCTION above) to train RL agents [48], [99], [127], [178], [214], [231], [248], [268], [271].”
Towards Causal Representation Learning (2021, background): “world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see INTRODUCTION above) to train RL agents [127, 248, 98, 47, 271, 178, 232, 214, 268].”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “Within MBRL, commonly explored methods include action-conditional, next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et…”
Shaping Belief States with Generative Environment Models for RL (2019, methodology): “This problem is characterized by many models’ failure to perform coherent long-term predictions, while performing accurate short-term predictions, even in trivial but partially observed environments [11, 13, 14].”
World Models (2018, methodology): “Subsequent works have used RNN-based models to generate many frames into the future (Chiappa et al., 2017; Oh et al., 2015; Denton & Birodkar, 2017), and also as an internal model to reason about the future (Silver et al.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “In [95], deep neural networks have been used to generate predictions in simulated environments over hundreds of time steps.”
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning (2017, methodology): “[5] introduces action-conditional dynamics parameterized with LSTMs and, as for the KVAE, a computationally efficient procedure to make long term predictions without generating high dimensional images at each time step.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Prior work on action-conditioned video prediction has studied predicting synthetic video game images [16], [29], 3D point clouds [19], and real-world images [30], [31], [32], using both direct autoregressive frame prediction [31], [32], [33] and latent variable models [34], [35].”
Unsupervised Learning of Disentangled Representations from Video (2017, background): “These can either be used to train action-conditional predictive models of future frames [2, 20] or inversekinematics models [1] which attempt to predict actions from current and future frame pairs.”
Stochastic Variational Video Prediction (2017, background): “One particularly common assumption is that the environment is deterministic and that there is only one possible future (Chiappa et al., 2017; Srivastava et al., 2015; Boots et al., 2014; Lotter et al., 2017).”