Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks

By Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang. 2019

286 citations recorded by the source corpus.

Vision-Language Navigation (VLN) is a task where an agent learns to navigate following a natural language instruction. The key to this task is to perceive both the visual scene and natural language sequentially. Conventional approaches fully exploit vision and language features in cross-modal grounding. However, the VLN task remains challenging, since previous works have implicitly neglected the rich semantic information contained in environments (such as navigation graphs or sub-trajectory semantics). In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to exploit the additional training signals derived from these semantic information. The auxiliary tasks have four reasoning objectives: explaining the previous actions, evaluating the trajectory consistency, estimating the progress and predict the next direction. As a result, these additional training signals help the agent to acquire knowledge of semantic representations in order to reason about its activities and build a thorough perception of environments. Our experiments demonstrate that auxiliary reasoning tasks improve both the performance of the main task and the model generalizability by a large margin. We further demonstrate empirically that an agent trained with self-supervised auxiliary reasoning tasks substantially outperforms the previous state-of-the-art method, being the best existing approach on the standard benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Robotics & Control

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “[59] 2020 Room-to-Room SPL, SR, OSR, TL, NE Zhu et al.”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, positive): “The ensemble achieves a success rate of 73%, which is 4 absolute percentage points greater than previously published work [28], and 2 absolute percentage points greater than concurrent, unpublished work [33].”
Language-Conditioned Imitation Learning for Robot Manipulation Tasks (2020, background): “Beyond robotics, the combination of vision and language has received ample attention in visual question-and-answering systems (VQA) [22, 5] and vision-and-language navigation (VNL) [34, 18, 10].”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, methodology): “[Zhu et al., 2019] Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang.”
Vision-Dialog Navigation by Exploring Cross-Modal Memory (2020, methodology): “Rich information is explored by several self-supervised auxiliary reasoning tasks [38] to improve the visual grounding during navigation.”
VLP: A Survey on Vision-language Pre-training (2022, unknown): “Vision-Language Navigation (VLN) [96, 97, 98].”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “Among all tasks, visual navigation has been most studied in the vision and robotics community [4, 13, 21, 40, 76, 77, 94, 95].”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “AuxRN [60] applies several auxiliary losses to learn comprehensive representations, Qi et al.”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “In these years, 3D vision has gained significant attention and development, driven by the rising popularity of autonomous driving [11, 69, 71], navigation [72, 76, 98], 3D scene understanding [2, 43, 46, 74], and robotics [31, 67].”
FILM: Following Instructions in Language with Modular Methods (2021, background): “A plethora of works have been published on embodied vision and language tasks, such as VLN (Anderson et al., 2018b; Fried et al., 2018; Zhu et al., 2020), Embodied Question Answering (Das et al., 2018; Gordon et al., 2018), and topics of multimodal representation learning (Wang et al., 2020; Bisk…”
Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View (2020, background): “Compared to interior navigation in the Room-to-Room (R2R) task, the Touchdown task is much harder: e.g. the current state-of-the-art success rate (equivalent to TC) for R2R on the validation unseen dataset is 55% (Zhu et al., 2019).”
Episodic Transformer for Vision-and-Language Navigation (2021, background): “Most recent VLN approaches rely on recurrent architectures [39, 60, 68, 73] where the internal state is expected to keep information about previous actions and observations.”
Dense Regression Network for Video Grounding (2020, background): “This task has drawn increasing attention over the past few years due to its vast potential applications in video understanding [38, 3, 45, 44, 6], video retrieval [42, 8], and human-computer interaction [35, 20, 50], etc.”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “…of research in VLN utilizes soft attention over individual words for cross-modal grounding in both the natural language instruction and the visual scene (Wang et al., 2018, 2019; Tan et al., 2019; Landi et al., 2019; Xia et al., 2020; Wang et al., 2020b,a; Xiang et al., 2020; Zhu et al., 2020b).”