Learning to Act by Predicting the Future

By Alexey Dosovitskiy, Vladlen Koltun. 2016

292 citations recorded by the source corpus.

We present an approach to sensorimotor control in immersive environments. Our approach utilizes a high-dimensional sensory stream and a lower-dimensional measurement stream. The cotemporal structure of these streams provides a rich supervisory signal, which enables training a sensorimotor control model by interacting with the environment. The model is trained using supervised learning techniques, but without extraneous supervision. It learns to act based on raw sensory input from a complex three-dimensional environment. The presented formulation enables learning without a fixed goal at training time, and pursuing dynamically changing goals at test time. We conduct extensive experiments in three-dimensional simulations based on the classical first-person game Doom. The results demonstrate that the presented approach outperforms sophisticated prior formulations, particularly on challenging tasks. The results also show that trained models successfully generalize across environments and goals. A model trained using the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which was held in previously unseen environments.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Reinforcement Learning

Citations in the corpus

Recurrent World Models Facilitate Policy Evolution (2018, neutral): “This has been demonstrated not only in early work [58, 85] (when compute was a million times more expensive than today) but also in recent studies [13] on several competitive VizDoom environments.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, background): “The state representation could also be made more efﬁ-cient by exploiting auxiliary tasks such as pixel control or feature control (Jaderberg et al. 2016), supervised predictions (Dosovitskiy and Koltun 2016) or successor features (Kulkarni et al. 2016b).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “A number of such methods have been explored in the conventional online RL setting (Tamar et al., 2016; Dosovitskiy and Koltun, 2016; Oh et al., 2017), and in the offline RL setting, such techniques have been applied effectively to learning navigational policies for mobile robots from previously coll”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A non-exhaustive list includes pixel control [55], contrastive predictive coding (CPC) [56], action-conditional CPC [20], frame reconstruction [35, 57], next-frame prediction [18, 58, 20, 59] and successor representations [60, 61, 62].”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, background): “They have scored commanding successes in the venues of strategic games [5,7, 58,79], image classification [78], face [80,81] recognition, speech recognition [82] and e.”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “For generalization experiments, we pre-train on a different map with different random textures from (Dosovitskiy & Koltun, 2016) and each episode lasts for 2100 time steps.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Experiments show that the BDFP outperforms the DFP in most cases, classic navigation approaches generally outperform learning-based ones with RGB-D inputs.”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, background): “An important distinction between our approach and that of [7] is that our model is conditioned on a sequence of actions, which is critical for off-policy training.”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “While several works augmented IL with goal conditioning (Dosovitskiy & Koltun, 2016; Codevilla et al., 2018), these goals must be speciﬁed in advance during training, and are typically simple ( e.g. , turning left or right).”
End-to-End Driving Via Conditional Imitation Learning (2017, methodology): “Assume that each observation o = (cid:104) i , m (cid:105) comprises an image i and a low-dimensional vector m that we refer to as measurements, following Dosovitskiy and Koltun [9].”
Emergence of Grounded Compositional Language in Multi-Agent Populations (2017, methodology): “To help policy training avoid local minima in more complex environments, we found it helpful to include auxiliary goal prediction tasks, similar to recent work in reinforcement learning (Dosovitskiy and Koltun 2016; Silver et al. 2016).”
CARLA: An Open Urban Driving Simulator (2017, methodology): “The agent reached a score of 65 out of 100 after 10 million simulation steps – a good result compared to 60 out of 100 reported by Dosovitskiy and Koltun [5] after 50 million simulation steps for A3C with less optimized hyperparameters.”
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments (2017, methodology): “Further details are provided by Dosovitskiy and Koltun (2017).”
On Evaluation of Embodied Navigation Agents (2018, background): “The agent does not carry internal state across time steps and does not construct an internal representation of its environment [9, 32].”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, background): “Many variants of goal-conditioned value functions have been proposed in the literature Foster & Dayan (2002); Sutton et al. (2011); Schaul et al. (2015); Dosovitskiy & Koltun (2016).”
Semi-parametric Topological Memory for Navigation (2018, methodology): “DL-based approaches to navigation can be broadly divided into three classes: purely reactive (Dosovitskiy & Koltun, 2017; Zhu et al., 2017), based on unstructured general-purpose memory such as LSTM (Mnih et al., 2016; Mirowski et al., 2017), and employing a navigation-speciﬁc memory structure…”
Visual Semantic Planning Using Deep Successor Representations (2017, methodology): “To address the second and third challenges, we cast VSP as a policy learning problem, typically tackled by reinforcement learning [11, 16, 22, 30, 35, 46].”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “…connected to learning to predict the future via “successor” representations (Dayan, 1993; Kulka-rni et al., 2016b; Barreto et al., 2017) or forecasts (Schaul & Ring, 2013; Lample & Chaplot, 2017; Dosovitskiy & Koltun, 2017) and are trained to be predictive of features extracted from future states.”