REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments

By Yuankai Qi, Qi Wu, Peter Anderson, X. Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel. 2019

506 citations recorded by the source corpus.

One of the long-term challenges of robotics is to enable robots to interact with humans in the visual world via natural language, as humans are visual animals that communicate through language. Overcoming this challenge requires the ability to perform a wide variety of complex tasks in response to multifarious instructions from humans. In the hope that it might drive progress towards more flexible and powerful human interactions with robots, we propose a dataset of varied and complex robot tasks, described in natural language, in terms of objects visible in a large set of real images. Given an instruction, success requires navigating through a previously-unseen environment to identify an object. This represents a practical challenge, but one that closely reflects one of the core visual problems in robotics. Several state-of-the-art vision-and-language navigation, and referring-expression models are tested to verify the difficulty of this new task, but none of them show promising results because there are many fundamental differences between our task and previous ones. A novel Interactive Navigator-Pointer model is also proposed that provides a strong baseline on the task. The proposed model especially achieves the best performance on the unseen test split, but still leaves substantial room for improvement compared to the human performance. Repository: https://github.com/YuankaiQi/REVERIE.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “However, our findings are also relevant for similar datasets such as Touchdown (Chen et al., 2019; Mehta et al., 2020), CVDN (Thomason et al., 2019), REVERIE (Qi et al., 2020), and the multilingual Room-acrossRoom (RxR) dataset (Ku et al., 2020).”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Substantial modeling [50] and benchmark [61] efforts in agent navigation focus on identifying object landmarks [12] and destinations [7].”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “Datasets have been collected for both indoor (Anderson et al., 2018b; Thomason et al., 2019b; Qi et al., 2020) and outdoor (Chen et al., 2019; Mehta et al., 2020) environments; success in these is based on clearly-defined, objective task completion rather than language or vision specific…”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “The richness and generalizability of natural language makes it an effective medium for directing mobile agents in navigation tasks, even in environments they have never encountered before (Anderson et al., 2018b; Chen et al., 2019; Misra et al., 2018; de Vries et al., 2018; Qi et al., 2019).”
AllenAct: A Framework for Embodied AI Research (2020, background): “…including visual navigation [12, 13, 14], question answering [15, 16], task completion [17], instruction following [18, 19], language grounding [20, 21], grasping [22, 23], object manipulation [24, 25], future prediction [26], and multi-agent collaboration [27, 28]; as well as using interaction as…”
SQA3D: Situated Question Answering in 3D Scenes (2022, background): “Notable tasks including robotic navigation (Das et al., 2018; Anderson et al., 2018; Savva et al., 2019; Chen et al., 2019; Wijmans et al., 2019b; Qi et al., 2020; Deitke et al., 2022) and vision-based manipulation (Kolve et al.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “SimilartoR2R,the REVERIE dataset [107] is also built based on the Matterport3D simulator.”
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, methodology): “We follow [37] to adopt previous state-of-the-art VLN approaches and apply visual grounding models on the end viewpoint during inference to complete REVERIE problem.”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, background): “A great variety of scenarios have been set up for relevant research, such as long range indoor and street view navigation with comprehensive instructions [3, 5, 17], communication based visual navigation [7, 25, 34], and navigation for object localization and visual question answering [6, 29].”
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling (2019, unknown): “Vision-and-Language Navigation Navigation in 3D environments based on natural language instruction has recently been investigated by many studies [3,8,21,22,26,20,35,25,11,33,32,16,29,34].”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, methodology): “Hence, all navigation tasks defined in this paper are novel to NavGPT. of modules to achieve such a goal (Anderson et al. 2018; Qi et al. 2020b; Krantz et al. 2020; Ku et al. 2020; He et al. 2021; Gu et al. 2022; Zhu et al. 2022; Hong et al. 2020a, 2022; Zhao, Qi, and Wu 2023; Qiao et al. 2023b),…”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “Training instruction-following navigation agents has attracted increasing research attention [1, 2, 6–8 , 29].”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, background): “[52] study referring expressions in an embodied setting, where semantic annotations are projected from 3D to 2D bounding boxes on images observed by an agent.”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, background): “Prior work also investigates navigation with target descriptions in supervised settings [42, 60, 87].”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “2, at each time step, the network takes four sets of tokens as input; the previous state token st ́1, the language tokensX , the visual tokens V t for scene, and the visual tokensOt for objects (only in REVERIE [44]).”