Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation

By Vihan Jain, Gabriel Ilharco, Alexander Ku, Ashish Vaswani, Eugene Ie, Jason Baldridge. 2019

231 citations recorded by the source corpus.

Advances in learning and representations have reinvigorated work that connects language to other modalities. A particularly exciting direction is Vision-and-Language Navigation(VLN), in which agents interpret natural language instructions and visual scenes to move through environments and reach goals. Despite recent progress, current research leaves unclear how much of a role language under-standing plays in this task, especially because dominant evaluation metrics have focused on goal completion rather than the sequence of actions corresponding to the instructions. Here, we highlight shortcomings of current metrics for the Room-to-Room dataset (Anderson et al.,2018b) and propose a new metric, Coverage weighted by Length Score (CLS). We also show that the existing paths in the dataset are not ideal for evaluating instruction following because they are direct-to-goal shortest paths. We join existing short paths to form more challenging extended paths to create a new data set, Room-for-Room (R4R). Using R4R and CLS, we show that agents that receive rewards for instruction fidelity outperform agents that focus on goal completion.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, background): “…navigation tests the agent’s ability to take action according to human instructions, which recently gains popularity in embodied AI (Anderson et al., 2018b; Chen et al., 2019; Jain et al., 2019; Chen et al., 2019; Qi et al., 2020b; Krantz et al., 2020; Nguyen and Daumé III, 2019; Ku et al., 2020).”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, background): “Most recent efforts (e.g., Fu et al., 2019; Huang et al., 2019; Jain et al., 2019; Wang et al., 2019, etc.) have used the Room-to-Room (R2R) dataset (Anderson et al., 2018b), which contains 4675 unique paths in the train split, 340 in the valseen split (same environments, new paths), and an…”
Vision-and-Dialog Navigation (2019, neutral): “Using the whole navigation path, P ∗, as supervision rather than only the end node has been considered in other work [32].”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “Thomason et al. (2019a) showed that agents can exploit effective priors over R2R paths, and Jain et al. (2019) showed that R2R paths encourage goal seeking p0 p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13r0 r1 r2 r3 (a) Panorama navigation graph with room annotations. p0 p1 p2 p3 p4 p5…”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, neutral): “[10].”
VALAN: Vision and Language Agent Navigation (2019, background): “Metrics for assessing agents performance include: Navigation Error (NE) which is a measure of the distance between agent’s stopping location and the goal; Success Rate (SR) measures how often the agent stops within some acceptable distance of the goal; Success weighted by Path Length (SPL) [2] measu”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “• R4R [3] extends R2R dataset by concatenating two adjacent tail-to-head trajectories in R2R.”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, methodology): “On the R4R dataset (Table 2), our method significantly outperforms the previous state-of-the-arts over all the metrics on the validation unseen split.”
Effective and General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping (2019, methodology): “Despite its simplicity, we show in Section 4 that AD correlates surprisingly well with human judgments and even outperforms CLS (but not nDTW).”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “(5) Coverage weighted by Length Score (CLS) (Jain et al., 2019): Similar to nDTW, this metric also encourages path fidelity.”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, methodology): “We expand this data by concatenating consecutive segments to create more challenging instructions, including with exploration challenges [32].”
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling (2019, unknown): “Vision-and-Language Navigation Navigation in 3D environments based on natural language instruction has recently been investigated by many studies [3, 8, 21, 22, 26, 20, 33, 25, 11, 32, 31, 16].”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “Inlanguage-guidedgrasping,semanticscanoriginatefrom explicitinstructions[166],[167]andimplicitinstructions[163], [164].”
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021, background): “The VLN benchmark is further extended to study the fidelity of instruction following (Jain et al., 2019) and examined to understand the bias of the benchmark (Zhang et al., 2020).”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “…AI (EAI) has garnered substantial attention, and notable advancements have been made in various tasks (Anderson et al., 2018; Krantz et al., 2020; Jain et al., 2019; Savva et al., 2019; Deitke et al., 2020; Weihs et al., 2021; Ehsani et al., 2021; Shridhar et al., 2020; Pad-makumar et al., 2022;…”