OpenMask3D: Open-Vocabulary 3D Instance Segmentation
By Ayca Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, Francis Engelmann. Neural Information Processing Systems, 2023
344 citations recorded by the source corpus.
We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/1a262a8eaf86f850b1b268dd40ce089ed88f339d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Andrea Vedaldi
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
Citations in the corpus
- ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “Where [66] assumes a pre-built point cloud map of the scene, [67] builds a map on the go.”
- Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “Existing methods [8, 37] rely on manually-labeled datasets, which are both costly and limited in scope, or requires accurately scanned 3D point clouds [33, 42] as input.”