Language and Visual Entity Relationship Graph for Agent Navigation

By Yicong Hong, Cristian Rodriguez-Opazo, Yuankai Qi, Qi Wu, Stephen Gould. 2020

161 citations recorded by the source corpus.

Vision-and-Language Navigation (VLN) requires an agent to navigate in a real-world environment following natural language instructions. From both the textual and visual perspectives, we find that the relationships among the scene, its objects,and directional clues are essential for the agent to interpret complex instructions and correctly perceive the environment. To capture and utilize the relationships, we propose a novel Language and Visual Entity Relationship Graph for modelling the inter-modal relationships between text and vision, and the intra-modal relationships among visual entities. We propose a message passing algorithm for propagating information between language elements and visual entities in the graph, which we then combine to determine the next action to take. Experiments show that by taking advantage of the relationships we are able to improve over state-of-the-art. On the Room-to-Room (R2R) benchmark, our method achieves the new best performance on the test unseen split with success rate weighted by path length (SPL) of 52%. On the Room-for-Room (R4R) dataset, our method significantly improves the previous best from 13% to 34% on the success weighted by normalized dynamic time warping (SDTW). Code is available at: this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Robotics & Control, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Robotics & Control

In Signal Processing

Citations in the corpus

VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “[17] propose a graph network to model the intra- and inter-modal relationships among the contextual and visual clues.”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, background): “[Hong et al., 2020] proposed a language and visual entity relationship graph modeling relations among scene, object and directional clues.”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, methodology): “For long instructions, NavGPT can break it down with sub-goals, similar to previous works on fine-graining R2R data (Hong et al. 2020b; He et al. 2021; Zhao et al. 2022), and plan to reach the destination by effectively identifying landmarks from observations, similar to works on utilizing objects…”
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, methodology): “Numerous methods [47, 48, 19, 7, 12, 10, 46, 36, 17, 20] have been proposed to address the VLN task.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “VLN is an essential task, aiming at navigating in unseen environments following linguistic instructions, as shown in Fig.”