ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language

By Dave Zhenyu Chen, Angel X. Chang, Matthias Nießner. European Conference on Computer Vision, 2019

616 citations recorded by the source corpus.

We introduce the task of 3D object localization in RGB-D scans using natural language descriptions. As input, we assume a point cloud of a scanned 3D scene along with a free-form description of a specified target object. To address this task, we propose ScanRefer, learning a fused descriptor from 3D object proposals and encoded sentence embeddings. This fused descriptor correlates language expressions with geometric features, enabling regression of the 3D bounding box of a target object. We also introduce the ScanRefer dataset, containing 51,583 descriptions of 11,046 objects from 800 ScanNet scenes. ScanRefer is the first large-scale effort to perform object localization via natural language expression directly in 3D.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6c161841cdb547f77930942e4ab46f4369751676

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

OpenScene: 3D Scene Understanding with Open Vocabularies (2022, background): “These works have yielded network architectures and training protocols that have significantly pushed the boundary of several 3D scene understanding benchmarks, including 3D object classification [56], 3D object detection and localization [4, 6, 15, 49], 3D semantic and instance segmentation [2, 3, 5”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, neutral): “Speciﬁcally, we study two variants of EQA under a common umbrella: episodic memory (EM-EQA) and active exploration (A-EQA), depending on the agent platform.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “Embodied interaction includes not only question-answering, but also performing tasks like grasping and placing objects based on human instructions.”
Deep Multimodal Data Fusion (2024, background): “…[30, 204, 217], RGB-depth scene classification task [222], multimodal recommendation task [193, 194], neuroimaging-based disease classification task [158], 3D object localization with natural language descriptions [21], and object segmentation in the 3D scenes according to the query sentences [66].”
GRES: Generalized Referring Expression Segmentation (2023, background): “, Scanrefer [2] focuses on 3D objects and Clevrtex [19] focuses on unsupervised learning.”
SQA3D: Situated Question Answering in 3D Scenes (2022, methodology): “For the zero-shot models, we extract 3D scene captions from two sources: ScanRefer (Chen et al., 2020) and ReferIt3D (Achlioptas et al.”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, methodology): “This additional set of data will be particularly useful when comparing our method to the Unique setting of [18] (Table 4), since it assumes that the target object is the only instance of that class in the scene.”
ScanQA: 3D Question Answering for Spatial Scene Understanding (2021, methodology): “First, we automatically generate question-answer pairs from the captions to identify objects in 3D scenes drawn from the ScanRefer dataset [10].”
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation (2021, background): “More recently, meth-ods that use text to localize objects in 3D scene have been explored [2,6].”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Furthermore, while this work focuses on editable view synthesis, it is also intriguing to transfer DFF to other applications, including 3D registration of text queries [16, 1, 47, 4] or robot teaching [32, 80].”
An Embodied Generalist Agent in 3D World (2023, neutral): “It’s a kitchen for cooking. they fall short in embodied tasks, e.g ., acting within 3D environments.”
3D-LLM: Injecting the 3D World into Large Language Models (2023, background): “ScanQA [45] requires a model to answer questions related to the 3D world; ScanRefer [5] asks a model to localize a region that the text expression refer to; 3D captioning [8] tests models’ abilities to generate captions describing the 3D scenes.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, neutral): “Answer: While chasing the baseball, the player may inadvertently collide with other players.”