Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

By Xin Eric Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-fang Wang, William Yang Wang, Lei Zhang. 2018

642 citations recorded by the source corpus.

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms previous methods by 10% on SPL and achieves the new state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7% to 11.7%).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “Such data augmentation is so effective that it is nearly ubiquitous in the best performing agents (Wang et al., 2019; Huang et al., 2019; Li et al., 2019).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “…power of neural architectures, various works looked into creating instruction-following agents that can perform manipulation (Lynch & Sermanet, 2020; 2021), navigation (Fried et al., 2018; Wang et al., 2019; Majumdar et al., 2020), or both (Suglia et al., 2021; Hill et al., 2020; Fu et al., 2019).”
LILA: Language-Informed Latent Actions (2021, background): “A variety of methods have sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for reinforcemen”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “We note that a given pre-built grid-map of receptacle locations is a strong prior assumption, but future work could incorporate existing models from the vision-language navigation literature (Anderson et al., 2018a; Wang et al., 2019) for map-free navigation.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “Existing beam-search [17, 47, 52] and backtracking solutions [24, 28] are infeasible due to the larger action and state spaces, long horizon, and inability to undo certain actions.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “Human-generated natural language instructions are used in [MacMahon et al., 2006; Bisk et al., 2016; Misra et al., 2017; Janner et al., 2018; Chen et al., 2018; Anderson et al., 2018; Goyal et al., 2019; Wang et al., 2019].”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, methodology): “Several existing papers [11, 30] and one concurrent hitherto-unpublished work [10] consider path-instruction compatibility as an auxiliary loss function or reward for VLN agents.”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, background): “Vision and language navigation (VLN) models learn to follow natural language directions that are longer that those considered in this work Misra et al. [2017, 2018], Anderson et al. [2018], Fried et al. [2018], Wang et al. [2019], Zhu et al. [2019].”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, methodology): “…this has 1https://github.com/google-research-datasets/RxR led to a focus on data augmentation (Fried et al., 2018; Tan et al., 2019), pre-training (Wang et al., 2019; Huang et al., 2019; Li et al., 2019), multi-task learning (Wang et al., 2020) and better generalization through piece-wise…”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, background): “• RCM+SIL( TRAIN ) (Wang et al., 2019): an agent trained with cross-modal grounding locally and globally via reinforcement learning.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “However, prior work has often studied 2D environments [24, 32, 27, 52] and simplified actuators [48, 18, 12].”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, neutral): “NE Random Student-forcing [1] (CVPR 2018) RPA [2] (ECCV 2018) Speaker-Follower [3] (NeurIPS 2018) RCM [4] (CVPR 2019) Self-Monitoring (ICLR 2019) Regretful [5] (CVPR 2019) Evaluation metrics: • Navigation Error (NE): mean of the shortest path distance between the agent’s final position and the goal ”
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, background): “Further, co-attention mechanisms become an essential tool in many vision-language applications and sequential modeling tasks, such as visual question answering [39,42,76,79], visual dialog [74,85], vision-language navigation [66], and video segmentation [40,60], showing its effectiveness in capturin”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “Comparison with SoTA Results in Table 1 compare the single-run (greedy search, no pre-exploration [55]) performance of different agents on the R2R benchmark.”