Vision-and-Dialog Navigation

By Jesse Thomason, Michael Murray, Maya Cakmak, Luke S. Zettlemoyer. 2019

405 citations recorded by the source corpus.

Robots navigating in human environments should use language to ask for assistance and be able to understand human responses. To study this challenge, we introduce Cooperative Vision-and-Dialog Navigation, a dataset of over 2k embodied, human-human dialogs situated in simulated, photorealistic home environments. The Navigator asks questions to their partner, the Oracle, who has privileged access to the best next steps the Navigator should take according to a shortest path planner. To train agents that search an environment for a goal location, we define the Navigation from Dialog History task. An agent, given a target object and a dialog history between humans cooperating to find that object, must infer navigation actions towards the goal in unexplored environments. We establish an initial, multi-modal sequence-to-sequence model and demonstrate that looking farther back in the dialog history improves performance. Sourcecode and a live interface demo can be found at https://cvdn.dev/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “, 2019), a simulated house (Anderson et al., 2018; Thomason et al., 2019b; Shridhar et al., 2020), or the real”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “However, our findings are also relevant for similar datasets such as Touchdown (Chen et al., 2019; Mehta et al., 2020), CVDN (Thomason et al., 2019), REVERIE (Qi et al., 2020), and the multilingual Room-acrossRoom (RxR) dataset (Ku et al., 2020).”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “Since the proposal of R2R [3], researchers have dramatically improved the navigation performance of models [17, 24, 28, 52, 53] with techniques like progress monitoring [27], as well as introduced task variants with additional, on-route instructions [37, 38, 50].”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, neutral): “video understanding [26], navigation assistants [27–29].”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “icable to most existing approaches to VLN, leading to our agent PREVALENT. We consider three VLN scenarios as downstream tasks: Room-to-room (R2R) [3], cooperative vision-and-dialog navigation (CVDN) [30], and “Help, Anna!” (HANNA) [22]. The overall pre-training and ﬁnetuning pipeline is shown in Fi”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “In vision-dialog navigation [59], the Cooperative Vision-and-Dialog Navigation (CVDN) [105] dataset is used.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “, navigation [76, 75], multi-agent communication [34], image captioning [58, 6, 59], etc.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, methodology): “In this way, EmBERT can be applied to other language-guided tasks such as VLN and Cooperative Vision-and-Dialog Navigation (Thomason et al., 2019).”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “A number of VLN datasets situated in photorealistic 3D reconstructions of real locations contain human instructions or dialogue: R2R (Anderson et al., 2018b), Touchdown (Chen et al., 2019; Mehta et al., 2020), CVDN (Thomason et al., 2019b) and REVERIE (Qi et al., 2020).”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “CVDN (Thomason et al., 2019) contains natural conversations in which a human assistant aids another human in navigation tasks but offers limited language interaction simulation, as language assistance is not available when the agent deviates from the collected trajectories and tasks.”
Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks (2019, methodology): “Various associated tasks have been proposed, including Visual Question Answering (VQA) [1], Visual Dialog Answering [38], Vision-Language Navigation (VLN) [5] and Visual Com-monsense Reasoning (VCR) [44].”
VALAN: Vision and Language Agent Navigation (2019, background): “VALAN implements a variant of SEED RL/IMPALA [10] to address the scalability challenges inherent in VLN/VDN datasets.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “4) Modular Approaches: F3RM [166] seeks to elevate CLIP’s text-image priors into 3-”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “CVDN75 (Thomason et al., 2019) is a dataset76 of embodied, human-human dialogs situated in a simulated, photorealistic home environment.”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “• CVDN [9] defines a navigation from dialog history task, which requires an agent to arrive at goal regions based on multi-turn question-answering dialogs.”
Learning Rewards from Linguistic Feedback (2020, background): “…et al. 2011; Djalali, Lauer, and Potts 2012; Potts 2012; Ilinykh, Zarrieß, and Schlangen 2019; Suhr et al. 2019), reference pragmatics (He et al. 2017; Udagawa and Aizawa 2019), navigation (Thomason et al. 2019), and “Wizard of Oz” experiments (Kim et al. 2009; Allison, Luger, and Hofmann 2018).”
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments (2019, background): “…Navigation TouchDown [5] Nav-Instruction Detailed Dynamic Navigation VLNA [24], HANNA[23] Nav-Dialog High Dynamic Find Object TtW [8] Nav-Dialog High Dynamic Navigation CVDN [25] Nav-Dialog High Dynamic Find Room ReferCOCO [30] RefExp an object in an image given a natural language expression.”
Cognitive Architectures for Language Agents (2023, background): “…2023; Nguyen et al., 2022b; 2019; Nguyen and Daumé III, 2019) or clarification (Biyik and Palan, 2019; Sadigh et al., 2017; Padmakumar et al., 2022; Thomason et al., 2020; Narayan-Chen et al., 2019) – or entertain or emotionally help people (Zhang et al., 2020; Zhou et al., 2018; Pataranutaporn et…”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “Vision-Language Navigation deﬁnes the task of requiring an agent to navigate through a visual environment based on natural language instructions (Anderson et al., 2018b; Misra et al., 2018; Chen et al., 2019; Jain et al., 2019; Nguyen and Daumé III, 2019; Thomason et al., 2020).”