Language-Grounded Indoor 3D Semantic Segmentation in the Wild

By Dávid Rozenberszki, Or Litany, Angela Dai. European Conference on Computer Vision, 2022

313 citations recorded by the source corpus.

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on available datasets. However, current 3D semantic segmentation benchmarks contain only a small number of categories -- less than 30 for ScanNet and SemanticKITTI, for instance, which are not enough to reflect the diversity of real environments (e.g., semantic image understanding covers hundreds to thousands of classes). Thus, we propose to study a larger vocabulary for 3D semantic segmentation with a new extended benchmark on ScanNet data with 200 class categories, an order of magnitude more than previously studied. This large number of class categories also induces a large natural class imbalance, both of which are challenging for existing 3D semantic segmentation methods. To learn more robust 3D features in this context, we propose a language-driven pre-training method to encourage learned 3D features that might have limited training examples to lie close to their pre-trained text embeddings. Extensive experiments show that our approach consistently outperforms state-of-the-art 3D pre-training for 3D semantic segmentation on our proposed benchmark (+9% relative mIoU), including limited-data scenarios with +25% relative mIoU using only 5% annotations.

Subfields: Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/b3dd7c78ae3960474541f01d9a9ce7b5de65c110

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Point Transformer V3: Simpler, Faster, Stronger (2023, methodology): “8, we present PTv3’s validation results on the ScanNet v2 [17] and ScanNet200 [67] instance segmentation benchmarks.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, methodology): “The most closely related work to ours of this type is Rozenberszki et al. [46], since they use the CLIP embedding to pre-train a model for 3D semantic segmentation.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “On the other hand, ongoing studies on zero-shot perception in 3D still suffer from the lack of effective, efficient, and high-resolution architectures and large-scale annotated datasets [54, 33, 29, 101, 78, 30].”
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes (2023, methodology): “More recently, the ScanNet200 benchmark [34] was proposed on top of the ScanNet dataset for recognition of 200 annotated classes.”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “We inspected every sequence (and each frame therein) over the ScanNet validation set, and identified the following sequences as being at least the scale of a one-bedroom apartment, and not suffering motion blur: scene0011, scene0050, scene0231, scene0378, scene0518.”
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “ScanNet200 allows to test an algorithm’s performance under the natural imbalance of classes, particularly for challenging long-tail classes such as coffee-kettle and pottedplant.”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2023, methodology): “It has been employed in various downstream applications, from image-based object detection and segmentation [13, 14] to 3D applications [15, 16].”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “We perform our experiments using the 312 validation scenes from the ScanNet200 dataset [54], and evaluate for 3D instance segmentation task using the closed vocabulary of 200 categories from the ScanNet200 [54] annotations.”