OpenScene: 3D Scene Understanding with Open Vocabularies

By Songyou Peng, Kyle Genova, ChiyuMaxJiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas A. Funkhouser. Computer Vision and Pattern Recognition, 2022

718 citations recorded by the source corpus.

Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries. For example, to perform SOTA zero-shot 3D semantic segmentation it first infers CLIP features for every 3D point and later classifies them based on similarities to embeddings of arbitrary class labels. More interestingly, it enables a suite of open-vocabulary scene understanding applications that have never been done before. For example, it allows a user to enter an arbitrary text query and then see a heat map indicating which parts of a scene match. Our approach is effective at identifying objects, materials, affordances, activities, and room types in complex 3D scenes, all using a single model trained without any labeled 3D data.

Subfields: Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/774408d8848b129d93fb67548ec6571d99b31a2d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “…the 2D representations produced by image and language foundation models to the 3D world and show impressive results on open-vocabulary tasks, including language-guided object grounding [17], [18], [24], [26], [36], 3D reasoning [37], [38], robot manipulation [39]–[41] and navigation [42], [43].”
LERF: Language Embedded Radiance Fields (2023, background): “VL-Maps [18] and Open-Scene [27] build a 3D volume of language features which can be queried for navigation tasks, by using pre-trained pixel-aligned language encoders [14, 21].”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “More recently, OpenScene [48] applied pixel-aligned LSeg [24] and OpenSeg [18] to open-set 3D segmentation; demonstrating that features from these 2D extractors can be distilled into neural networks operating over 3D data.”
An Embodied Generalist Agent in 3D World (2023, neutral): “See the results in Appendix I.4.”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “9 Open-vocabulary OpenScene [49] (2D Fusion) - OpenSeg [18] 11.”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “Existing methods [8, 37] rely on manually-labeled datasets, which are both costly and limited in scope, or requires accurately scanned 3D point clouds [33, 42] as input.”