Tactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation

By Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, Siddhartha S. Srinivasa. 2019

185 citations recorded by the source corpus.

We present the Frontier Aware Search with backTracking (FAST) Navigator, a general framework for action decoding, that achieves state-of-the-art results on the 2018 Room-to-Room (R2R) Vision-and-Language navigation challenge. Given a natural language instruction and photo-realistic image views of a previously unseen environment, the agent was tasked with navigating from source to target location as quickly as possible. While all current approaches make local action decisions or score entire trajectories using beam search, ours balances local and global signals when exploring an unobserved environment. Importantly, this lets us act greedily but use global signals to backtrack when necessary. Applying FAST framework to existing state-of-the-art models achieved a 17% relative gain, an absolute 6% gain on Success rate weighted by Path Length.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In LLM Agents & Tool Use

Citations in the corpus

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “Existing beam-search [17, 47, 52] and backtracking solutions [24, 28] are infeasible due to the larger action and state spaces, long horizon, and inability to undo certain actions.”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “A novel neural decoding scheme was proposed in [12] with search, to balance global and local information.”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, methodology): “In this setting, a high-probability path is typically generated by performing beam search through the environment and ranking paths according to either: (1) their probability under a ‘follower’ model [3,8,13,20,21,30,31], as in Figure 1 (left), or (2) by how well they explain the instruction accordi”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, methodology): “• FAST (Ke et al., 2019): an agent which combines global and local knowledge to compare partial trajectories of different lengths, enabling efficient backtrack after a mistake.”
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling (2019, background): “Vision-and-Language Navigation Navigation in 3D environments based on natural language instruction has recently been investigated by many studies [3, 8, 21, 22, 26, 20, 33, 25, 11, 32, 31, 16].”
Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks (2019, unknown): “FAST [49] 5.”
FILM: Following Instructions in Language with Modular Methods (2021, methodology): “For Visual Language Navigation, which is the most comparable to the setting of our work, methods with impressive performances (Ke et al., 2019; Wang et al., 2019; Ma et al., 2019) have been proposed since the introduction of R2R (Anderson et al., 2018b).”
Chasing Ghosts: Instruction Following as Bayesian State Tracking (2019, background): “Other work has focused on developing modules for estimating progress towards the goal [4] and learning when to backtrack [5, 8].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2019) ResNet-152 LSTM BackTranslation RMSprop 3 (Ke et al., 2019) LSTM TacticalRewind 7”
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, background): “1, where we mainly compare to the RCM [47], SelfMonitor [33], FAST [24], and REVERIE [37].”
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (2018, unknown): “, Matterport: (Ma et al., 2019a; Ke et al., 2019; Wang et al., 2019; Ma et al., 2019b; Tan et al., 2019; Fried et al., 2018), and EQA: (Das et al.”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “FAST [19] resorts self-correction navigation, while APS [12] samples adversarial paths for training to enhance the model’s ability to generalise.”
Visual Landmark Selection for Generating Grounded and Interpretable Navigation Instructions (2019, background): “Signiﬁcant efforts have been devoted to the problem of following natural language navigation instructions in perceptually realistic virtual environments [1, 5, 18, 11, 10, 8, 15].”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, methodology): “Later, the Self-Monitoring agent [22] applies a visual-textual co-grounding network and a progress monitor to guide the transition of language attention, this work is further improved by path-scoring and backtracking methods, such as the Regretful agent [23] and the Tactical Rewind [18].”