Learning to Navigate in Complex Environments

By Piotr Wojciech Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu and 2 more. 2016

958 citations recorded by the source corpus.

Learning to navigate in complex environments with dynamic elements is an important milestone in developing AI agents. In this work we formulate the navigation question as a reinforcement learning problem and show that data efficiency and task performance can be dramatically improved by relying on additional auxiliary tasks leveraging multimodal sensory inputs. In particular we consider jointly learning the goal-driven reinforcement learning problem with auxiliary depth prediction and loop closure classification tasks. This approach can learn to navigate from raw sensory input in complicated 3D mazes, approaching human-level performance even under conditions where the goal location changes frequently. We provide detailed analysis of the agent behaviour, its ability to localise, and its network activity dynamics, showing that the agent implicitly learns key navigation abilities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Finally, it is often useful to learn multiple general value functions (Sutton et al., 2011; Mankowitz et al., 2018; Lample & Chaplot, 2016; Mirowski et al., 2017; Jaderberg et al., 2017b) or options (Sutton et al.”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Concurrently, a suite of methods combining model-free deep reinforcement learning with auxiliary tasks has shown large beneﬁts on a wide variety of domains (Jaderberg et al., 2016; van den Oord et al., 2018; Mirowski et al., 2017).”
Shaping Belief States with Generative Environment Models for RL (2019, background): “The challenge of model-based RL in rich 3D environments has compelled some researchers to use privileged information such as camera-locations [15], depth information [53], and other ground-truth state variables of the state simulator [54, 49].”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “Deep reinforcement learning has demonstrated success in policy search for tasks in domains like game playing (Mnih et al., 2015; Silver et al., 2016; 2017; Kempka et al., 2016; Mirowski et al., 2017) and robotic control (Levine et al., 2016a;b; Pinto & Gupta, 2016).”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “There are several prior works that use reinforcement learning to navigate in 3D environments from pixel inputs or playing ATARI games such as (Mirowski et al., 2017; Mnih et al., 2015; 2016), but they rely on intermediate external rewards provided by the environment.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “The vast majority of work on this topic has considered hand engineered auxiliary tasks, such as inferring the depth map from the RGB observation, detection of loop closures (Mirowski et al., 2016), pixel control (Jaderberg et al.”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, background): “RL-based approaches for learning goal-directed navigation have been proposed [29, 30], but typically learn using hand-crafted reward signals that are difficult to calculate in the real-world.”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, background): “…demonstrate strong navigation performance in the indoor environments, the recently proposed deep reinforcement learning (DRL) based methods [1], [2], [3], [4] open up a new possibility that avoids the complexity of constructing these sophisticated SLAM pipelines and directly learns model-free…”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “For example, in navigation games (Mirowski et al., 2016; Lowe et al., 2017; Wu et al., 2018), the reward is typically set to the negative distance from the target location L T to the agent’s location L A plus a success bonus, so the feature vector φ ( s, a ) can be written as a 2-dimensional vector…”
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, background): “Another common best-practice in RL are auxiliary tasks Mirowski et al. (2016); Jaderberg et al. (2016), in which the agent produces extra output-heads that are trained on supervised tasks and optimized alongside the RL loss.”
Stratospheric Aerosol Injection as a Deep Reinforcement Learning Problem (2019, methodology): “We pre-train the encoder on ImageNet (Mirowski, 2016) and fine-tune the output layers on 2000 output samples of HadCM3 rollouts based on aerosol density distributions drawn randomly from a 73- dimensional Dirichlet distribution with shape parameters αk = 1.5 (to discourage extremes) and output…”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “, 2015), and navigating complex environments (Zhu et al., 2017; Mirowski et al., 2016).”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “This procedure is expected to be beneﬁcial in environments with sparse rewards, where unsupervised modelling losses can complement return maximization as learning target as recently explored in Jaderberg et al. [38] and Mirowski et al. [39].”
Distral: Robust multitask reinforcement learning (2017, methodology): “Possible directions of future research include: combining Distral with techniques which use auxiliary losses [12, 15, 14], exploring use of multiple distilled policies or latent variables in the distilled policy to allow for more diversity of behaviours, exploring settings for continual learning…”
DeepMind Lab (2016, methodology): “A I] 1 3 D ec 2 01 it has been used to develop asynchronous methods for reinforcement learning (Mnih et al., 2016), unsupervised auxiliary tasks (Jaderberg et al., 2016), and to study navigation (Mirowski et al., 2016).”