Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation

By Xin Eric Wang, Wenhan Xiong, Hongmin Wang, William Yang Wang. 2018

225 citations recorded by the source corpus.

Existing research studies on vision and language grounding for robot navigation focus on improving model-free deep reinforcement learning (DRL) models in synthetic environments. However, model-free DRL models do not consider the dynamics in the real-world environments, and they often fail to generalize to new scenes. In this paper, we take a radical approach to bridge the gap between synthetic studies and real-world practices---We propose a novel, planned-ahead hybrid reinforcement learning model that combines model-free and model-based reinforcement learning to solve a real-world vision-language navigation task. Our look-ahead module tightly integrates a look-ahead policy model with an environment model that predicts the next state and the reward. Experimental results suggest that our proposed method significantly outperforms the baselines and achieves the best on the real-world Room-to-Room dataset. Moreover, our scalable method is more generalizable when transferring to unseen environments, and the relative success rate is increased by 15.5% on the unseen test set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “For training objective, we experiment with the Self-Critical Sequence Training (SCST) in Rennie et al. (2017), where CIDEr (Vedantam et al., 2015) metric is optimized using REINFORCE algorithm (Williams, 1992).”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “Since the proposal of R2R [3], researchers have dramatically improved the navigation performance of models [17, 24, 28, 52, 53] with techniques like progress monitoring [27], as well as introduced task variants with additional, on-route instructions [37, 38, 50].”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “• RPA [33]: an agent that combines model-free and model-based reinforcement learning, using a look-ahead module for planning.”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “Recent advancements at the intersection of Computer Vision, Natural Language Processing and Robotics have tried to bring together the understanding of the visual world and that of natural language from the perspective of robots and embodied agents [1], [2], [3], [4], [5].”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “Prior work has reported higher performance on the seen rather than unseen environments [1, 53], illustrating the issue of generalizing to new environments.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, methodology): “[43] introduced a hybrid approach that combines model-free and model-based reinforcement learning (RL) to improve the model’s generalizability.”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, background): “…beam search through the environment and ranking paths according to either: (1) their probability under a ‘follower’ model [3,8,13,20,21,30,31], as in Figure 1 (left), or (2) by how well they explain the instruction according to a ‘speaker’ (instruction generation) model [8, 28], as in…”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, methodology): “Recently, Wang et al. (2018a) combines purely supervised learning and RL training however, they use a different algorithm named MIXER (Ranzato et al., 2015), which computes cross entropy (XE) losses for the first k actions and RL losses for the remaining. from the unpaired Y′ using the backward…”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, methodology): “Our approach takes into account the agent’s past and future behavior to determine actions that are most and least beneficial to them, combining the advantages of both model-based and progress-estimating methods (Wang et al., 2018; Ma et al., 2019a,b).”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, methodology): “• RPA (Wang et al., 2018): is an agent which combines model-free and model-based reinforcement learning, using a look-ahead module for planning.”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “Table 3: Performance comparison with the state of arts without beam search : Student-forcing (Anderson et al., 2018b), RPA (Wang et al., 2018b), and Speaker-Follower (Fried et al., 2018).”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “Previous study shows the essentials of modules to achieve such a goal [2, 46, 29, 30, 22, 19, 60, 72, 23, 25], whereas a large number of research reveal the crucial effect of training strategies [59, 53].”
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling (2019, background): “Vision-and-Language Navigation Navigation in 3D environments based on natural language instruction has recently been investigated by many studies [3, 8, 21, 22, 26, 20, 33, 25, 11, 32, 31, 16].”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, methodology): “The task has been extensively studied in [Wang et al., 2018; Fried et al., 2018; Tan et al., 2019; Huang et al., 2019; Wang et al., 2019; Hong et al., 2021; Chen et al., 2021].”
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, unknown): “visual-language navigation [1, 41, 33], embodied question answering [13, 19] and other tasks.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “It is then improved by exploiting hierarchical architecture (Yu et al., 2017) and then further optimized by incorporating reinforcement learning with rewards (Wang et al., 2018a) for generating relevant and expressive narrative paragraphs.”
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, methodology): “Numerous methods [47, 48, 19, 7, 12, 10, 46, 36, 17, 20] have been proposed to address the VLN task.”