Learning Exploration Policies for Navigation

By Tao Chen, Saurabh Gupta, Abhinav Kumar Gupta. 2019

260 citations recorded by the source corpus.

Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that the use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks. Code and Videos are available at: this https URL.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “In classical robotics, exploration is done through passive or active simultaneous localisation and mapping (SLAM) [47, 53] to build a map of the environment.”
Neural Topological SLAM for Visual Navigation (2020, methodology): “The results indicate that NTS performs better than both end-to-end RL based baselines [9,47] and methods using metric spatial maps [6, 9].”
Cross-View Semantic Segmentation for Sensing Surroundings (2019, unknown): “t learning. Given that current graphics simulation engines can render realistic scenes, recognition algorithms can be trained on data pulled from simulation engines (i.e. for visual navigation models [14, 49, 41, 9, 21]). Several techniques have been proposed to address the domain adaptation issue [”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, methodology): “These policies have been trained with reinforcement learning to maximize coverage, predicting semantically rich regions, intrinsic rewards [13, 14, 18, 19], or by using planning in conjunction with latent variable and affordance models [17, 20, 21].”
Spatial Action Maps for Mobile Manipulation (2020, background): “These works focus almost exclusively on how to best train neural networks for the task, for example using deep reinforcement learning [19, 47, 6], supervised [34, 12] and self-supervised learning [25], or predicting the future [9].”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, methodology): “This replaces the traditional area-coverage reward used in ANS and other current models [11,10,52], which rewards the increment in the actual area seen, not the correctly registered area in the map.”
An Exploration of Embodied Visual Exploration (2020, unknown): “First, prior work evaluates on different simulation environments such as SUN360 [42, 49], ModelNet [42, 49], VizDoom [39, 45], SUNCG [14], DeepMindLab [45], and Matterport3D [13].”
Embodied Semantic Scene Graph Generation (2021, methodology): “PPO+Map[32] The agent utilizes Resnet-18 to encode RGB frame, local map and global map respectively and concatenates them as the input for the LSTM to predict the next action.”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, background): “Recent work investigates learned alternatives for exploration [7,14,56,57, 63].”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform (2020, background): “Spatial navigation approaches [62, 17, 64, 23, 25, 56, 79, 49, 30, 14, 10] typically address navigating towards a pre-specified coordinate or a frame of a scene and they focus on understanding the geometry of the scene and learning better exploration strategies.”
Semantic Curiosity for Active Visual Learning (2020, unknown): “27 Coverage Exploration [14] 3.”
Learning to plan with uncertain topological maps (2020, unknown): “to visit as much space as possible as quickly as possible similar to [10,11].”
Visual Room Rearrangement (2021, background): “…tasks have been proposed in this domain: navigation towards objects [4, 51, 48, 7] or towards a speciﬁc point [1, 38, 47, 36], scene exploration [9, 8], embodied question answering [18, 13], task completion [55], instruction following [2, 41], object manipulation [16, 52], multi-agent…”
Electronic skins and machine learning for intelligent soft robots (2020, background): “(66) showed a direct policy learning algorithm with spatial memory and bootstrapped with human-mediated imitation learning without explicit task rewards.”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, background): “For single tasks, methods that leverage some task-specific knowledge in learning have been shown to improve performance [10, 88, 38] in realistic environments.”
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation (2020, background): “[10] focus on building a task-agnostic exploration policy and demonstrate that this helps for downstream navigation tasks.”
Learning to Visually Navigate in Photorealistic Environments Without any Supervision (2020, unknown): “[6] propose a coverage reward that encourages the agent to explore every part of its latent map.”