Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

By Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, Stefan Lee. 2020

578 citations recorded by the source corpus.

We develop a language-guided navigation task set in a continuous 3D environment where agents must execute low-level actions to follow natural language navigation directions. By being situated in continuous environments, this setting lifts a number of assumptions implicit in prior work that represents environments as a sparse graph of panoramas with edges corresponding to navigability. Specifically, our setting drops the presumptions of known environment topologies, short-range oracle navigation, and perfect agent localization. To contextualize this new task, we develop models that mirror many of the advances made in prior settings as well as single-modality baselines. While some of these techniques transfer, we find significantly lower absolute performance in the continuous setting -- suggesting that performance in prior `navigation-graph' settings may be inflated by the strong implicit assumptions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, background): “…navigation tests the agent’s ability to take action according to human instructions, which recently gains popularity in embodied AI (Anderson et al., 2018b; Chen et al., 2019; Jain et al., 2019; Chen et al., 2019; Qi et al., 2020b; Krantz et al., 2020; Nguyen and Daumé III, 2019; Ku et al., 2020).”
Language-Conditioned Imitation Learning for Robot Manipulation Tasks (2020, background): “Beyond robotics, the combination of vision and language has received ample attention in visual question-and-answering systems (VQA) [22, 5] and vision-and-language navigation (VNL) [34, 18, 10].”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “…these can help with agent training, but they also open up new possibilities for studying grounded language pragmatics in the VLN setting, and for training VLN agents with perspective cameras – either in the graph-based simulator or by lifting RxR into a continuous simulator (Krantz et al., 2020).”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “While these approaches have been largely studied in simulated spatial games [19, 21, 22, 23] or in object-directed visual navigation in simulated robots [24, 25, 26, 27, 28, 29, 23] some of which include high-level object interaction [30], in this work we focus on the domain of learning control for ”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, background): “Future work could extend our history-aware transformer to VLN with continuous actions [54] and could benefit from pretraining on larger navigation datasets.”
Visual Language Maps for Robot Navigation (2022, background): “Recently, also Vision-andLanguage Navigation (VLN) has received increased attention [8], [22].”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “Hence, all navigation tasks defined in this paper are novel to NavGPT. of modules to achieve such a goal (Anderson et al. 2018; Qi et al. 2020b; Krantz et al. 2020; Ku et al. 2020; He et al. 2021; Gu et al. 2022; Zhu et al. 2022; Hong et al. 2020a, 2022; Zhao, Qi, and Wu 2023; Qiao et al. 2023b),…”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “For instance, in embodied AI, the houses may be used for navigation [58, 89, 118, 132, 117, 128, 74, 130], multi-agent interaction [51, 52, 2], rearrangement and interaction [115, 36, 39, 23, 106], manipulation [33, 86, 32, 122], Sim2Real transfer [27, 54, 66], embodied vision-andlanguage [105, 87, ”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “VLN-CE [105] extends R2R and R4R to continuous environments, embodied agents can move freely in the scene.”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “Meanwhile, vision-language navigation (VLN) (Anderson et al., 2018; Jain et al., 2019; Krantz et al., 2020) augments visual observation with natural language descriptions, enabling the agent to plan a sequence of actions based on a comprehensive understanding of multiple modalities to successfully…”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, background): “Prior work investigates language-based navigation, where language provides step-by-step instructions for the task [2,34,39,40,71].”
Curious Representation Learning for Embodied Intelligence (2021, methodology): “We report SPL, Success and goal distance on the val-unseen split in [26], corresponding to unseen rooms, and report results on val-seen setting in the appendix.”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, unknown): “• CMA [40] employs cross-modal attention to integrate instructions with RGB observations for action prediction.”
Topological Planning with Transformers for Vision-and-Language Navigation (2020, methodology): “For evaluation, we use the VLN-CE dataset [29] and evaluate on environments which were seen (val-seen) and unseen (val-unseen) during training.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “To verify the effectiveness of such ability of our model, we conduct our experiments on the VLN-CE benchmark [100], demanding the agent to function in a continuous environment.”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “Future Work As suggested by the significant improvement on R2R [3] achieved by our VLNœ BERT, we expect that the model can improve the performance of other navigation settings such as street navigation [6] and navigation in continuous environments [22].”