Visual Representations for Semantic Target Driven Navigation

By Arsalan Mousavian, Alexander Toshev, Marek Fiser, Jana Kosecka, James Davidson. 2018

235 citations recorded by the source corpus.

What is a good visual representation for navigation? We study this question in the context of semantic visual navigation, which is the problem of a robot finding its way through a previously unseen environment to a target object, e.g. go to the refrigerator. Instead of acquiring a metric semantic map of an environment and using planning for navigation, our approach learns navigation policies on top of representations that capture spatial layout and semantic contextual cues. We propose to use semantic segmentation and detection masks as observations obtained by state-of-the-art computer vision algorithms and use a deep network to learn the navigation policy. The availability of equitable representations in simulated environments enables joint training using real and simulated data and alleviates the need for domain adaptation or domain randomization commonly used to tackle the sim-to-real transfer of the learned policies. Both the representation and the navigation policy can be readily applied to real non-synthetic environments as demonstrated on the Active Vision Dataset [1]. Our approach successfully gets to the target in 54% of the cases in unexplored environments, compared to 46% for a non-learning based approach, and 28% for a learning-based baseline.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “They show that this method outperforms a learned LSTM policy without a map [94] in previously unseen environments.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Embodied Navigation Agent Navigation in 3D environments [50, 25, 26, 14] is an essential capability of a mobile intelligent system that functions in the physical world.”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, neutral): “…(Das et al., 2018; Gordon et al., 2018), and navigation (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Zhu et al., 2017; de Vries et al., 2018; Yuke Zhu, 2017; Mousavian et al., 2018; Wayne et al., 2018; Wang et al., 2018a; Mirowski et al., 2017; 2018; Zamir et al., 2018).”
Learning Object Relation Graph and Tentative Policy for Visual Navigation (2020, background): “[18] fuse semantic segmentation and detection masks and then feed the fused features into their policy network for navigation.”
Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation (2019, background): “Semantic Target Driven Navigation [27] learns policy only based on detection and segmentation result, so that model can be easily transferred to unseen environments.”
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation (2019, unknown): “[13] demonstrate the use of semantic segmentation and detection masks for target driven visual navigation.”
Cross-View Policy Learning for Street Navigation (2019, background): “The research on artificial agent navigation can be applied to real world domains ranging from the neuroscience of grid and place cells in mammals [3, 9] to the autonomy of indoor and outdoor mobile robots [49, 39, 31, 38, 45, 32].”
Object Goal Navigation using Goal-Oriented Semantic Exploration (2020, methodology): “How do we design a computational model for building an episodic memory and using it effectively based on semantic priors for efficient navigation in unseen environments? One popular approach is to use end-to-end reinforcement or imitation learning with recurrent neural networks to build episodic mem”
Exploiting Scene-specific Features for Object Goal Navigation (2020, methodology): “erform path planning on explicit maps [13][4][16]. More recently, however, approaches based on Reinforcement Learning have been presentedthroughpolicies basedon RecurrentNeuralNetworks[17][15][22][10][20]. Mirowski et al. [17] deﬁne an approach that jointly learns the goal-driven Reinforcement Learn”
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks (2019, methodology): “[30] use an LSTM policy for target driven navigation.”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, background): “, 2019] and [Mousavian et al., 2019] showed that the semantic segmentation and the depth map can help improve the generalization ability.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “Furthermore, several studies [12], [50], [51] utilize semantic segmentation as an additional representation for navigational purposes.”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks (2018, methodology): “Self-supervised approaches in particular have been used to reduce sample complexity for active tasks [16, 34, 29, 15].”
Simultaneous Mapping and Target Driven Navigation (2019, methodology): “Following the work of Mousavian et al [15], we train our policy model to predict the cost of each action a at a certain state s and a given target c using an L1 loss:”
Learning Navigation Behaviors End-to-End With AutoRL (2018, background): “Deep learning for navigation: Machine learning for navigation has seen a recent boom, particularly in point-to-point navigation with vision [29], [37].”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, background): “They leverage the data either by making some task-agnostic assumption about the data distribution being simpler than the raw pixels (unsupervised approaches like the autoencoder family [34, 83, 41, 48] and Generative Adversarial Networks [77, 23, 57, 15]) or they exploit some known structure (so-cal”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, background): “More recent work uses deep learning to learn navigation [79,20,56,41,77,75,59,64] or exploration [48,6,57,28,51] policies in an end-to-end fashion.”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform (2020, background): “Semantic navigation approaches [22, 75, 69, 71, 39, 52, 41] attempt to learn the semantics of the target in conjunction with navigation.”