Action-Conditional Video Prediction using Deep Networks in Atari Games

By Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, Satinder Singh. 2015

909 citations recorded by the source corpus.

Motivated by vision-based reinforcement learning (RL) problems, in particular Atari games from the recent benchmark Aracade Learning Environment (ALE), we consider spatio-temporal prediction problems where future (image-)frames are dependent on control variables or actions as well as previous frames. While not composed of natural scenes, frames in Atari games are high-dimensional in size, can involve tens of objects with one or more objects being controlled by the actions directly and many other objects being influenced indirectly, can involve entry and departure of objects, and can involve deep partial observability. We propose and evaluate two deep neural network architectures that consist of encoding, action-conditional transformation, and decoding layers based on convolutional neural networks and recurrent neural networks. Experimental results show that the proposed architectures are able to generate visually-realistic frames that are also useful for control over approximately 100-step action-conditional futures in some games. To the best of our knowledge, this paper is the first to make and evaluate long-term predictions on high-dimensional video conditioned by control inputs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “Generative modeling of video holds promise for applications such as content creation, forecasting, transfer learning and model-based reinforcement learning (Srivastava et al., 2015; Carl Vondrick, 2016; Oh et al., 2015; Kaiser et al., 2019).”
Grounded Language Learning in a Simulated 3D World (2017, background): “The objective is to predict the visual environment v t +1 conditioned on the prior visual input v t and the action a t (Oh et al., 2015).”
Dialog-based Language Learning (2016, methodology): “Finally, forward prediction models, which we make use of in this work, have been used for learning eye tracking [21], controlling robot arms [12] and vehicles [26], and action-conditional video prediction in atari games [16, 23].”
Video Pixel Networks (2016, neutral): “Current approaches range from mean squared error models based on deep neural networks (Srivastava et al., 2015a; Oh et al., 2015), to models that predict quantized image patches (Ranzato et al., 2014), incorporate motion priors (Patraucean et al., 2015; Finn et al., 2016) or use adversarial losses…”
Conditional Image Generation with PixelCNN Decoders (2016, background): “While such unconditional models are fascinating in their own right, many of the practical applications of image modelling require the model to be conditioned on prior information: for example, an image model used for reinforcement learning planning in a visual environment would need to predict futur”
Deep multi-scale video prediction beyond mean square error (2015, methodology): “Oh et al. (2015) deﬁned an action conditional auto-encoder model to predict next frames of Atari-like games.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “Oh et al. (2015) and Chiappa et al. (2017) show that learning predictive models of Atari 2600 environments is possible using appropriately chosen deep learning architectures.”
Learning Latent Dynamics for Planning from Pixels (2018, background): “…et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018), and pure video prediction without planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016; Chiappa et al., 2017; Babaeizadeh et al., 2017; Gemici et al., 2017; Denton and Fergus,…”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “This weakness could be the reason that many previous works that learn dynamics models of RL environments do not actually use those models to fully replace the actual environments [8, 60].”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “We hypothesize this is because ACVP is a much larger network optimized for long term predictions, and does not take advantage of skip-connections and the discriminator loss as in GDM .”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “Most previous work on learning simulators for ALE games apply deterministic models, and do not consider learning state-space models for efﬁcient Monte-Carlo rollouts (Oh et al., 2015).”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, methodology): “Unfortunately, to date, it has proven difficult to learn models for such domains with sufficient fidelity to realise the benefits of look-ahead planning (Oh et al., 2015; Talvitie, 2017).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Learning generative models is a very challenging task (Bellemare et al., 2013, 2014; Oh et al., 2015; Chiappa et al., 2017) and so far, there has been no clear demonstration of successful planning with a learned model in the ALE.”
Value Prediction Network (2017, background): “Another line of work [23, 4, 8, 30] uses observation-prediction models not for planning, but for improving exploration.”
The Predictron: End-To-End Learning and Planning (2016, background): “(Oh et al., 2015; Chiappa et al., 2016); but are yet to surpass state-of-the-art model-free methods in challenging RL domains with raw inputs (e.”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “nother (Barreto et al., 2016) or to different scales of reward (Kulkarni et al., 2016). Another, related line of work involves learning models of the environment (Schmidhuber, 2010; Xie et al., 2015; Oh et al., 2015). Although learning environment models as auxiliary tasks could improve RL agents (e”
Value Iteration Networks (2016, background): “conditioning the kernels on the input (Oh et al., 2015) may provide an additional flexibility.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, methodology): “Oh et al., 2015), its count-based nature results in extremely fast learning, making it an appealing candidate for exploration.”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “Oh et al. [31] predict future frames in Atari games conditioned on the player’s action.”