Robust Navigation with Language Pretraining and Stochastic Sampling

By Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, Yejin Choi. 2019

132 citations recorded by the source corpus.

Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments. In this paper, we report two simple but highly effective methods to address these challenges and lead to a new state-of-the-art performance. First, we adapt large-scale pretrained language models to learn text representations that generalize better to previously unseen instructions. Second, we propose a stochastic sampling scheme to reduce the considerable gap between the expert actions in training and sampled actions in test, so that the agent can learn to correct its own mistakes during long sequential action decoding. Combining the two techniques, we achieve a new state of the art on the Room-to-Room benchmark with 6% absolute gain over the previous best result (47% -> 53%) on the Success Rate weighted by Path Length metric.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “Such data augmentation is so effective that it is nearly ubiquitous in the best performing agents (Wang et al., 2019; Huang et al., 2019; Li et al., 2019).”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “• PRESS [16]: an agent is trained with pre-trained language models and stochastic sampling to generalize well in the unseen environment.”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, methodology): “[17] incorporate language-only pretraining using a BERT model.”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “, 2019), pre-training (Wang et al., 2019; Huang et al., 2019; Li et al., 2019), multi-task learning (Wang et al.”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “Different methods have been adopted for VLN including DAgger [6, 24] and scheduled sampling [20, 25].”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “Recent work [11,16,21,23] employs BERT and its variants [7,20] to get better language understanding.”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “Other works improve vision and language representations (Hu et al., 2019; Li et al., 2019; Huang et al., 2019b,a; Hao et al., 2020; Majumdar et al., 2020) and propose an additional progress monitor module (Ma et al., 2019b,a; Ke et al., 2019) and object and action aware modules (Qi et al., 2020b)…”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, methodology): “Most previous agents proposed for the R2R navigation task are based on a sequence-to-sequence network [3] with grounding between vision and language [9, 21, 22, 32, 36].”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “Comparing to PRESS [29] and PREVALENT [14] which only fine-tune a pre-trained BERT for extracting language features, adding recurrence into V&L BERT and using the model directly as the navigator network allows the VLN learning to adequately benefit from the pre-trained knowledge.”