Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training

By Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, Jianfeng Gao. 2020

366 citations recorded by the source corpus.

Learning to navigate in a visual environment following natural-language instructions is a challenging task, because the multimodal inputs to the agent are highly variable, and the training data on a new task is often limited. In this paper, we present the first pre-training and fine-tuning paradigm for vision-and-language navigation (VLN) tasks. By training on a large amount of image-text-action triplets in a self-supervised learning manner, the pre-trained model provides generic representations of visual environments and language instructions. It can be easily used as a drop-in for existing VLN frameworks, leading to the proposed agent PREVALENT. It learns more effectively in new tasks and generalizes better in a previously unseen environment. The performance is validated on three VLN tasks. On the Room-to-Room benchmark, our model improves the state-of-the-art from 47\% to 51\% on success rate weighted by path length. Further, the learned representation is transferable to other VLN tasks. On two recent tasks, vision-and-dialog navigation and ``Help, Anna!'', the proposed PREVALENT leads to significant improvement over existing methods, achieving a new state of the art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “re is an growing interest in pre-training generic models to solve a variety of V+L problems, such as visual questionanswering (VQA), image-text retrieval and image captioningetc. The existing methods [38,39,23,5,47,36,19,10] employ BERT-like objectives [6] to learn Attention Interaction To further u”
Visual Instruction Tuning (2023, background): “For example, the vision-language navigation task [3, 19] and Habitat [47] require the embodied AI agent to follow natural language instructions and take a sequence of actions to complete goals in visual environments.”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Recent work has extended this to vision and language tasks [2, 45–50], leading to compelling results in Visual Question Answering [4], Commonsense Reasoning [51], Natural Language Visual Reasoning [52], Entailment [53], Image-Text Retrieval [54, 55], Referring Expressions [56], and Vision-Language N”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, background): “Several existing papers [11, 30] and one concurrent hitherto-unpublished work [10] consider path-instruction compatibility as an auxiliary loss function or reward for VLN agents.”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “…2022; Tan, Yu, and Bansal 2019; Parvaneh et al. 2020; Li and Bansal 2023), memory mechanism (Chen et al. 2021b; Pashevich, Schmid, and Sun 2021; Hong et al. 2023), pre-training (Hao et al. 2020; Chen et al. 2022a; Qiao et al. 2023a; Wang et al. 2023) have been adopted to alleviate data scarcity.”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “Other works improve vision and language representations (Hu et al., 2019; Li et al., 2019; Huang et al., 2019b,a; Hao et al., 2020; Majumdar et al., 2020) and propose an additional progress monitor module (Ma et al., 2019b,a; Ke et al., 2019) and object and action aware modules (Qi et al., 2020b)…”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, methodology): “On the validation unseen split, our method achieves very similar performance as the concurrent work PREVALENT [10], which exceeds previous methods on SR and SPL by a large margin.”
K-LITE: Learning Transferable Visual Models with External Knowledge (2022, methodology): “Hence, existing pre-trained models [49, 83, 58, 51, 81, 103, 38, 48, 30, 99, 77] would perform poorly on these knowledge-intensive V+L tasks [9].”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “The model initialised from PREVALENT [14], pre-trained especially for VLN, further improves the agent’s performance, achieving 63% SR (+8%) and 57% SPL (+5%) on the test unseen split3.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “• Action Prediction (AP) target at evaluating whether the agent developed for visionlanguage navigation (VLN) can select the right actions based on the current image and instruction [122].”
Kaleido-BERT: Vision-Language Pre-training on Fashion Domain (2021, methodology): “In contrast to the boom in generic tasks ( e.g ., VCR [9, 43, 78], VQA [32, 38]), other tasks such as visual relationship detection (RVL-BERT [10]), visual navigation ( i.e ., PERVALENT [24] and VLN-BERT [48]), and visual dialog ( e.g ., VisualD [50], VD-BERT [70]) are still in their infancy.”
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, methodology): “Numerous methods [47, 48, 19, 7, 12, 10, 46, 36, 17, 20] have been proposed to address the VLN task.”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “We compare HAMT with history input and PREVALENT [22] without history.”