Mask3D: Mask Transformer for 3D Semantic Instance Segmentation

By Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe. IEEE International Conference on Robotics and Automation, 2022

409 citations recorded by the source corpus.

Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques. Building on the successes of recent Transformer-based methods for object detection and image segmentation, we propose the first Transformer-based approach for 3D semantic instance segmentation. We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. In our model - called Mask3D - each object instance is represented as an instance query. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Combined with point features, the instance queries directly yield all instance masks in parallel. Mask3D has several advantages over current state-of-the-art approaches, since it neither relies on (1) voting schemes which require hand-selected geometric properties (such as centers) nor (2) geometric grouping mechanisms requiring manually-tuned hyper-parameters (e.g. radii) and (3) enables a loss that directly optimizes instance masks. Mask3D sets a new state-of-the-art on ScanNet test (+6.2mAP), S3DIS 6-fold (+10.1 mAP), STPLS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP).

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b798790509d52fcb9dd8b1aebdf24f7d6436e9c2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Robert W. Sumner
2. Marc Pollefeys
3. Fisher Yu
4. Lei Ke
5. Martin Danelljan
6. Francis Engelmann
7. Christian Häne
8. Andreas Geiger

In Computational Mechanics

1. Marc Pollefeys
2. Robert W. Sumner
3. Fisher Yu
4. Martin Danelljan
5. Francis Engelmann
6. Andreas Geiger
7. Matthias Nießner
8. Leonidas J. Guibas

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Martin Danelljan
4. Jitendra Malik
5. Dacheng Tao
6. Andrea Vedaldi
7. Philip H. S. Torr
8. Trevor Darrell

Citations in the corpus

• Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “Existing methods [8, 37] rely on manually-labeled datasets, which are both costly and limited in scope, or requires accurately scanned 3D point clouds [33, 42] as input.”
• Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “8(b) shows an overview”
• OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “The current state-of-the-art approach on the ScanNet200 benchmark [10, 57] is Mask3D [58], which leverages a transformer architecture to generate 3D mask proposals together with their semantic labels.”