Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

By Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian David Reid, Stephen Gould, Anton van den Hengel. 2017

1,852 citations recorded by the source corpus.

A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matter-port3D Simulator - a large-scale reinforcement learning environment based on real imagery [11]. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings - the Room-to-Room (R2R) dataset1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “An embodied agent, whether in a virtual world, such as a 2D Maze (MacMahon et al., 2006), a Baby AI in a grid world (Chevalier-Boisvert et al., 2019), Vision-and-Language Navigation (Anderson et al., 2018; Thomason et al., 2019b), or the real world (Tellex et al., 2011; Matuszek, 2018; Thomason et a”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “…summer research project [1] Since this now famously ambitious summer project, steady progress has been made towards systems that can demonstrate their visual understanding by generating or responding to natural language in the context of images, videos, or even full 3D environments [2–8].”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “These tasks have many potential applications, including robotics [5], document understanding [40], assisting visually-impaired people [7, 18], etc.”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “CLIP-ViL p with CLIP-Res50 outperforms models based on the widely used region-based encoder, BottomUp-TopDown (BUTD) ResNet101 (Anderson et al., 2018a).”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, background): “See Anderson et al. (2018a) and Ilharco et al. (2019) for details.”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “Shridhar et al., 2020a;b; Kolve et al., 2017; Savva et al., 2019; Anderson et al., 2018).”
LILA: Language-Informed Latent Actions (2021, background): “On the other end of the spectrum are more recent approaches in end-to-end robot learning for more complicated tasks spanning navigation [44, 51] and manipulation [55].”
Imitating Interactive Intelligence (2020, background): “Other work on language and interaction based in 3D simulated environments has focused on embodied instruction-following (Hill et al., 2019b), navigation (Anderson et al., 2017) or question-answering (Das et al., 2018).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “1 Unlike prior work on instruction following (MacMahon et al., 2006; Anderson et al., 2018a), which typically uses a ﬁxed corpus of cross-modal expert demonstrations, we argue that aligned parallel environments like ALFWorld offer a distinct advantage: they allow agents to explore, interact , and…”
Visual Instruction Tuning (2023, background): “For example, the vision-language navigation task [3, 19] and Habitat [47] require the embodied AI agent to follow natural language instructions and take a sequence of actions to complete goals in visual environments.”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[22] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel.”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “We consider three VLN scenarios as downstream tasks: Room-to-room (R2R) [3], cooperative vision-and-dialog navigation (CVDN) [30], and “Help, Anna!””
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “SceneNN [31] X S+I Stanford 2D-3D-S [8, 9, 75] X X S+I Matterport3D [6, 14, 57, 58, 75] X X S+I ScanNet [17] X X S+I Gibson [58, 75] X”
Embodied Question Answering (2017, background): “AI2-THOR [18], Matterport3D [31], Stanford 2D-3D-S [35]).”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “Recent advancements at the intersection of Computer Vision, Natural Language Processing and Robotics have tried to bring together the understanding of the visual world and that of natural language from the perspective of robots and embodied agents [1], [2], [3], [4], [5].”
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, methodology): “Additionally, the task formulations in CLiMB are mostly classification, but sequence-based vision-and-language tasks could allow the study of embodied navigation [Anderson et al., 2018] and task completion [Shridhar et al., 2020], and may be feasible in a more task-agnostic CLiMB framework.”
Continuous Scene Representations for Embodied AI (2022, background): “Recently, various embodied tasks such as navigation [5,8,37,54,56,63], instruction following [1,47], embodied question answering [15, 23], object manipulation [18,21,22,46], and room rearrangement [4,52] have been of interest.”