Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

By Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, Jason Baldridge. 2020

543 citations recorded by the source corpus.

We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations. We also provide results for a model that learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. The size, scope and detail of RxR dramatically expands the frontier for research on embodied language agents in simulated, photo-realistic environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “We apply our model to two vision-and-language navigation datasets: Room-to-Room (R2R, Anderson et al. (2018b)) and Room-across-Room (RxR, Ku et al. (2020)).”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “Visual Search cost ↓ measures the percentage of the available panoramic visual field that the annotator observes at each viewpoint, based on the pose traces provided by PanGEA and first proposed in the RxR dataset (Ku et al., 2020).”
LILA: Language-Informed Latent Actions (2021, methodology): “To do this, adding linguistically diverse data has been the gold standard [44, 51]; however, a new class of approaches have emerged that sidestep additional data requirements by tapping into the potential of pretrained language models [43, 50].”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[21] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “By operating only in simulation, our model also misses the full range of experience that can ground language in the world [11], such as haptic feedback during object manipulation [78, 79, 68], and audio [16] and speech [31, 41] features of the environment.”
Episodic Transformer for Vision-and-Language Navigation (2021, methodology): “To address this, the visual-and-language navigation (VLN) task is proposed to introduce rich and unstructured visual context for the agent to explore, perceive and execute upon [3, 13, 32, 33, 44].”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, background): “However, contemporary language-driven agents still require a large number of labeled examples (pairs of language instructions and gold trajectories) to learn each task, which is highly costly and hinders the development of truly versatile agents [34, 29, 25, 8, 37, 17, 40, 27, 11, 2, 16].”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “We carry out extensive experiments on various VLN tasks, including VLN with fine-grained instructions (R2R [6] and RxR [7]), high-level instructions (REVERIE [8] and our proposed R2R-Last), dialogs [9] as well as long-horizon VLN (R4R [3] and our proposed R2R-Back which requires the agent to return…”
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action (2022, unknown): “…approaches to VLN use a large dataset of simulated trajectories — over 1M demonstrations — annotated with ﬁne-grained language labels in indoor [1, 3–5, 22 ] Recent progress in using large-scale models of natural language and images trained on diverse data has enabled applications in a wide…”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “…tasks defined in this paper are novel to NavGPT. of modules to achieve such a goal (Anderson et al. 2018; Qi et al. 2020b; Krantz et al. 2020; Ku et al. 2020; He et al. 2021; Gu et al. 2022; Zhu et al. 2022; Hong et al. 2020a, 2022; Zhao, Qi, and Wu 2023; Qiao et al. 2023b), whereas a large…”
Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, background): “Instruction following tasks have been a popular testbed for language grounding (Chevalier-Boisvert et al., 2019; Anderson et al., 2018; Ku et al., 2020; Shridhar et al., 2020) prompting many improvements to decision making conditioned on language instructions (Yu et al., 2018; Lynch & Sermanet,…”
Mapping Language Models to Grounded Conceptual Spaces (2022, background): “We discuss the limitations of focusing on text-only inputs to teach models grounded concepts, as well as the implications of such work for allowing large language models to be mapped, in a data-efﬁcient way, to the grounded world.”
Language to Rewards for Robotic Skill Synthesis (2023, methodology): “One example is instruction following methods in navigation [30].”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, methodology): “We evaluate our agent on Room-to-Room (R2R) dataset (Anderson et al., 2018b) and Room-Across-Room (RxR) dataset (Ku et al., 2020).”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Moreover, the Waymo Open Sim Agents Challenge (WOSAC) [50,117] is the first public challenge to develop simulations with realistic and interactive agents.”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “Following these work, dialog-based navigation such as CVDN [52], VNLA [41] and HANNA [40], navigation for localising a remote object such as REVERIE [44], VLN in continuous environment [22], and multilingual navigation with spatialtemporal grounding such as RxR [24] have been proposed for further re”