Target-driven visual navigation in indoor scenes using deep reinforcement learning

By Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Kumar Gupta, Li Fei-Fei, Ali Farhadi. 2016

1,723 citations recorded by the source corpus.

Two less addressed issues of deep reinforcement learning are (1) lack of generalization capability to new goals, and (2) data inefficiency, i.e., the model requires several (and often costly) episodes of trial and error to converge, which makes it impractical to be applied to real-world scenarios. In this paper, we address these two issues and apply our model to target-driven visual navigation. To address the first issue, we propose an actor-critic model whose policy is a function of the goal as well as the current state, which allows better generalization. To address the second issue, we propose the AI2-THOR framework, which provides an environment with high-quality 3D scenes and a physics engine. Our framework enables agents to take actions and interact with objects. Hence, we can collect a huge number of training samples efficiently. We show that our proposed method (1) converges faster than the state-of-the-art deep reinforcement learning methods, (2) generalizes across targets and scenes, (3) generalizes to a real robot scenario with a small amount of fine-tuning (although the model is trained in simulation), (4) is end-to-end trainable and does not need feature engineering, feature matching between frames or 3D reconstruction of the environment.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “Navigation based simulator Our simulator is similar in spirit to recently proposed 3D RL environments based on game engines, such as ViZDoom [22], DeepMind Lab [7] and THOR [50].”
Embodied Question Answering (2017, background): “Visual navigation typically specifies agent goals either implicitly via the reward function [18, 19] (thus training a separate policy for each goal/reward), or explicitly by conditioning on goal state representations [23] like images of target objects [17, 24].”
Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Data (2017, background): “However Q-learning and DQN have demonstrated good performance in a lot of application that Markov (memoriless) property does not hold 24,25 .”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “6 steps, Zhu et al. (2017b) achieved 210 .”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, neutral): “Many types of goals have been used for this task, such as points, objects, images [78, 79] and areas [17].”
Dynamic Graph CNN for Learning on Point Clouds (2018, background): “ing due to efﬁciency considerations or instability of these techniques in the presence of noise. A few of the many recent applications of point cloud processing and analysis include indoor navigation [57], self-driving vehicles [33], robotics [40], and shape synthesis and modeling [14]. Modern appli”
The limits and potentials of deep learning for robotics (2018, background): “Partial success on transferring from simulation to a real robot has been reported [8], [48], [134], [138].”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, methodology): “When navigating in an environment without a map, the robot can either incrementally build a map while navigating around [21] or simply following a reactive setting to navigate based on current state and target state [1].”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, methodology): “For indoor navigation simulators like AI2-THOR [29], SUNCG [30] and Matterport3D [31] have emerged.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “…(RL) has been demonstrated to effectively learn a wide range of reward-driven skills, including playing games (Mnih et al., 2013; Silver et al., 2016), controlling robots (Gu et al., 2017; Schulman et al., 2015b), and navigating complex environments (Zhu et al., 2017; Mirowski et al., 2016).”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, methodology): “We used AI2-THOR for the problem of targetdriven navigation [23] using deep reinforcement learning and showed that the trained model can be transferred to real robots with some amount of fine-tuning.”
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (2016, background): “Consequently, there has been a renewed interest in training models in the synthetic domain and applying them in real–world settings [8, 47, 38, 42, 25, 32, 35, 37].”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “Inspired by this reasoning, recently there has been an increasing interest in more end-to-end learning-based approaches that go directly from pixels to actions (Zhu et al. 2017; Mnih et al. 2015; Levine et al. 2016) without going through explicit model or state estimation steps.”
IQA: Visual Question Answering in Interactive Environments (2017, methodology): “Map-less approaches [17, 22, 40, 60, 74] which use techniques such as obstacle avoidance and feature matching, depend upon implicit representations of the world to perform navigation, and lack long-term memory capabilities.”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “However, these works only evaluate the Environment 3D Realistic Large-scale Fast Customizable Atari (Bellemare et al., 2013) • OpenAI Universe (Shi et al., 2017) • • • Malmo (Johnson et al., 2016) • • • • DeepMind Lab (Beattie et al., 2016) • • • VizDoom (Kempka et al., 2016) • • • AI2-THOR (Zhu et ”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, background): “Zhu et al. (2017) address generalization of reinforcement learning to new target goals in visual search by providing the agent an image of the goal state.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Embodied Navigation Agent Navigation in 3D environments [50, 25, 26, 14] is an essential capability of a mobile intelligent system that functions in the physical world.”