Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention

By Khanh Nguyen, Debadeepta Dey, Chris Brockett, William B. Dolan. 2018

159 citations recorded by the source corpus.

We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level end-goals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress. To model language-based assistance, we develop a general framework termed Imitation Learning with Indirect Intervention (I3L), and propose a solution that is effective on the VNLA task. Empirical results show that this approach significantly improves the success rate of the learning agent over other baselines on both seen and unseen environments. Our code and data are publicly available at https://github.com/debadeepta/vnla .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “Since the proposal of R2R [3], researchers have dramatically improved the navigation performance of models [17, 24, 29, 53, 54] with techniques like progress monitoring [28], as well as introduced task variants with additional, on-route instructions [38, 39, 51].”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Concurrent to our work, [42, 24, 25, 26] studies the VLN tasks from various aspects, and [30] introduces a variant of the VLN task to find objects by requesting language assistance when needed.”
Vision-and-Dialog Navigation (2019, methodology): “Dataset —Language Context— —Visual Context— Human Amb UnderS Temporal Real-world Temporal Shared MARCO[8, 9], DRIF[10] 3 7 7 1I 7 Dynamic R2R[7], Touchdown[11] 3 7 7 1I 3 Dynamic EQA[12], IQA[13] 7 7 3 1Q 7 Dynamic CLEVR[14] 7 7 1Q 7 Static VQA[15, 16, 17] 3 7 1Q 3 Static CLEVR-Dialog[18] 7 7 2D 7 S”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, methodology): “Formally, I3L searches for policies that optimize: where L nav and L ask are the navigation and help-request loss functions, respectively, D env ˆ π ask is the environment distribution induced by ˆ π ask , and D state ˆ π nav , E is the state distribution induced by ˆ π nav in environment E .”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “The instruction-following literature includes several examples of language-based learning agents, centered on navigation [31], [32], [33], [10], [9], [34], [12], [35] or simple manipulation tasks [11], [36], [22].”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “Following these work, dialog-based navigation such as CVDN [52], VNLA [41] and HANNA [40], navigation for localising a remote object such as REVERIE [44], VLN in continuous environment [22], and multilingual navigation with spatialtemporal grounding such as RxR [24] have been proposed for further re”
Cognitive Architectures for Language Agents (2023, background): “Agents capable of generating language may ask for help (Ren et al., 2023; Nguyen et al., 2022b; 2019; Nguyen and Daumé III, 2019) or clarification (Biyik and Palan, 2019; Sadigh et al., 2017; Padmakumar et al., 2022; Thomason et al., 2020; Narayan-Chen et al., 2019) – or entertain or emotionally…”
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments (2019, background): “The most closely related challenge to that proposed here is that addressed in [23, 24, 25] whereby an agent must identify an object by requesting and interpreting natural language assistance.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “1), the ASKNAV49 (Nguyen et al., 2019) is built on top of Matterport3D50.”
Interactive Learning from Activity Description (2021, background): “The goal of a request-fulfilling agent is to map a given request in a situated environment to an execution that accomplishes the intent of the request (Winograd, 1972; Chen & Mooney, 2011; Tellex et al., 2012; Artzi et al., 2013; Misra et al., 2017; Anderson et al., 2018; Chen et al., 2019; Nguyen e”
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (2018, background): “Modern approaches use neural architectures to map natural language to the (simulated) world and execute actions (Paxton et al., 2019; Chen et al., 2018; Nguyen et al., 2018; Blukis et al., 2018; Fried et al., 2018; Mei et al., 2016).”