Visual Semantic Navigation using Scene Priors

By Wei Yang, X. Wang, Ali Farhadi, Abhinav Kumar Gupta, Roozbeh Mottaghi. 2018

379 citations recorded by the source corpus.

How do humans navigate to target objects in novel scenes? Do we use the semantic/functional priors we have built over years to efficiently search and navigate? For example, to search for mugs, we search cabinets near the coffee machine and for fruits we try the fridge. In this work, we focus on incorporating semantic priors in the task of semantic navigation. We propose to use Graph Convolutional Networks for incorporating the prior knowledge into a deep reinforcement learning framework. The agent uses the features from the knowledge graph to predict the actions. For evaluation, we use the AI2-THOR framework. Our experiments show how semantic knowledge improves performance significantly. More importantly, we show improvement in generalization to unseen scenes and/or objects. The supplementary video can be accessed at the following link: this https URL .

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “Recently, various embodied tasks such as navigation [5,8,37,54,56,63], instruction following [1,47], embodied question answering [15, 23], object manipulation [18,21,22,46], and room rearrangement [4,52] have been of interest.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Past work [98] that use human priors of knowledge integrated into a deep reinforcement learning framework has shown that artificial agent can tap onto human-like semantic/functional priors to aid the agent in learning to navigate and find unseen objects in the unseen environment.”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, methodology): “We used AI2-THOR for the problem of targetdriven navigation [26, 24, 22] using deep reinforcement learning and showed that the trained model can be transferred to real robots with some amount of fine-tuning.”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, methodology): “Both iTHOR and RoboTHOR have been used as platforms for the ObjectNav task [8,20,21].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, background): “…exploration, many navigation tasks are “goal-conditioned,” where the agent needs to navigate to a given position (i.e., point goal [58, 72, 74, 67, 11, 30]), a view of the environment (i.e., image goal [78, 43, 56]), or a specified object category (i.e., object goal [69, 10, 9, 75, 39, 1, 64]).”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, background): “For single tasks, methods that leverage some task-specific knowledge in learning have been shown to improve performance [10, 88, 38] in realistic environments.”
Object Goal Navigation using Goal-Oriented Semantic Exploration (2020, methodology): “[47] propose to incorporate semantic priors into a deep reinforcement learning framework by using Graph Convolutional Networks.”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform (2020, background): “Semantic navigation approaches [22, 75, 69, 71, 39, 52, 41] attempt to learn the semantics of the target in conjunction with navigation.”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, background): “…abstract entity variables in concrete objects: Prior works that model entities and their interactions often pre-specify the identity of the entities [24–30], provide additional supervision [31–34], or provide additional speciﬁcation such as segmentations [35], crops [36], or a simulator [37, 38].”
A Behavioral Approach to Visual Navigation with Graph Localization Networks (2019, background): “Other approaches have been tested in more realistic house or office settings [17, 18, 47, 50], but they ignore the collision problem entirely and allow the agents to continue their trajectories despite undergoing collisions.”
Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning (2018, background): “[46, 44] incorporate semantic knowledge to better generalize to unseen scenarios.”
Visual Room Rearrangement (2021, background): “Various tasks have been proposed in this domain: navigation towards objects [4, 51, 48, 7] or towards a speciﬁc point [1, 38, 47, 36], scene exploration [9, 8], embodied question answering [18, 13], task completion [55], instruction following [2, 41], object manipulation [16, 52], multi-agent…”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, methodology): “However, existing work relies heavily on model-based planners with the single-step dynamic model [5,11,48,67,74,80], lacking a trajectory-level optimization for long-horizon planning.”
Learning to Visually Navigate in Photorealistic Environments Without any Supervision (2020, neutral): “[34] extend this to navigation to an object described only by its name.”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks (2018, background): “ simple domains unrepresentative of the real world (e.g. ﬁxed tabletop domains). For single tasks, methods that leverage some task-speciﬁc knowledge in learning have been shown to improve performance [10,88,38] in realistic environments. Cognitive Psychology studies [44,73] suggest that one mechanis”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, background): “Learned approaches operating directly on pixels and/or depth as input can be robust to noise [11,10] and can generalize well on unseen environments [20,38,77,10] —even outperforming pure SLAM given sufficient experience [38].”
Exploiting Scene-specific Features for Object Goal Navigation (2020, methodology): “[34], instead, propose a Deep Reinforcement Learning model that exploits the relationships between objects, encoded through a Graph Convolutional Network, to incorporate semantic priors within the policy.”