Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

By Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan Al-Regib, Zsolt Kira, Richard Socher, Caiming Xiong. 2019

315 citations recorded by the source corpus.

The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our self-monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set). Code is available at this https URL .

Subfields: Aerospace Engineering, Computer Networks and Communications, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, neutral): “On R2R (Wang et al., 2018) 25 23 S-Follower (Fried et al., 2018) 35 28 RCM (Wang et al., 2019) 43 38 SMNA (Ma et al., 2019a) 48 35 Regretful (Ma et al., 2019b) 48 40 FAST-Short (Ke et al., 2019) 54 41 EnvDrop (Tan et al., 2019) 51 47 PRESS (Li et al., 2019b) 49 45 ALTR (Huang et al., 2019a) 48 45 CG”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “We use ALFRED to evaluate a sequence-to-sequence model with progress monitoring, shown to be effective in other vision-and-language navigation tasks [27].”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “• S MNA [18]: an agent trained with a visual-textual co-grounding module and a progress monitor on the panoramic action space.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Concurrent to our work, [42, 24, 25, 26] studies the VLN tasks from various aspects, and [30] introduces a variant of the VLN task to find objects by requesting language assistance when needed.”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, methodology): “In this setting, a high-probability path is typically generated by performing beam search through the environment and ranking paths according to either: (1) their probability under a ‘follower’ model [3,8,13,20,21,30,31], as in Figure 1 (left), or (2) by how well they explain the instruction accordi”
Skill Induction and Planning with Latent Language (2021, methodology): “…seq2seq , these are neural sequence-to-sequence models trained to map instructions to actions; they incorporate several standard modeling improvements from the instruction following literature, including progress monitoring (Ma et al., 2019) and pretrained object recognizers (Singh et al., 2020).”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, background): “Look Before You Leap (Wang et al., 2018a) 9.15 25.3 0.23 - - - - - - Speaker-Follower (Fried et al., 2018) 14.8 35.0 0.28 1257 53.5 0.01 - - - Self-Monitoring (Ma et al., 2019) 18.0 48.0 0.35 373 61.0 0.02 - - - Reinforced Cross-Modal (Wang et al., 2019) Implementation Details Similar to the traditi”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, methodology): “Our approach takes into account the agent’s past and future behavior to determine actions that are most and least beneﬁcial to them, combining the advantages of both model-based and progress-estimating meth-ods (Wang et al., 2018; Ma et al., 2019a,b).”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, methodology): “Recent work, e.g., using search strategies (Ke et al., 2019) or progress monitors (Ma et al., 2019b), has focused on mitigating the cost of computing top-k rollouts.”
The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation (2019, methodology): “Following [14, 12], we use a panoramic view as visual input.”
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, background): “1, where we mainly compare to the RCM [47], SelfMonitor [33], FAST [24], and REVERIE [37].”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform (2020, unknown): “Navigation using language instructions has been explored by [5, 68, 18, 31, 67, 36, 37].”
Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks (2019, methodology): “Our Step-wise MSE model performs 2.62% higher on the seen validation set 2.53% higher on the unseen validation set than Progress Monitor [23], indicating that label measured by normalized distances is noisier than label measured by steps.”
Vision-Dialog Navigation by Exploring Cross-Modal Memory (2020, methodology): “The self-monitoring method [21] is proposed to estimate progress made towards the goal.”
Chasing Ghosts: Instruction Following as Bayesian State Tracking (2019, background): “Other work has focused on developing modules for estimating progress towards the goal [4] and learning when to backtrack [5, 8].”
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021, methodology): “, 2018), the Self-Monitoring Navigation Agent(Ma et al., 2019a; Ke et al., 2019), the Regretful Agent (Ma et al.”
Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters (2019, methodology): “Recent works on VLN [11, 17, 18, 21] integrate the agent with a simplified action space in which it “only needs to make high-level decisions as to which navigable direction to go next” [11].”
Visual Landmark Selection for Generating Grounded and Interpretable Navigation Instructions (2019, background): “Significant efforts have been devoted to the problem of following natural language navigation instructions in perceptually realistic virtual environments [1, 5, 18, 11, 10, 8, 15].”