Self-Supervised Pretraining of 3D Features on any Point-Cloud

By Zaiwei Zhang, Rohit Girdhar, Armand Joulin, Ishan Misra. IEEE International Conference on Computer Vision, 2021

344 citations recorded by the source corpus.

Pretraining on large labeled datasets is a prerequisite to achieve good performance in many computer vision tasks like image recognition, video understanding etc. However, pretraining is not widely used for 3D recognition tasks where state-of-the-art methods train models from scratch. A primary reason is the lack of large annotated datasets because 3D data labelling is time-consuming. Recent work shows that self-supervised learning is useful to pretrain models in 3D but requires multi-view data and point correspondences. We present a simple self-supervised pretraining method that can work with single-view depth scans acquired by varied sensors, without 3D registration and point correspondences. We pretrain standard point cloud and voxel based model architectures, and show that joint pretraining further improves performance. We evaluate our models on 9 benchmarks for object detection, semantic segmentation, and object classification, where they achieve state-of-the-art results. Most notably, we set a new state-of-the-art for object detection on ScanNet (69.0% mAP) and SUNRGBD (63.5% mAP). Our pretrained models are label efficient and improve performance for classes with few examples.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/b7dfca646ac61ec85c8e015828a05c77f2d7d22f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Ishan Misra
2. Armand Joulin
3. Rohit Girdhar
4. Or Litany
5. Jie Zhou
6. Jiwen Lu
7. Yongming Rao
8. Jiageng Mao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Yu Qiao
5. Hongsheng Li
6. Yi Yang
7. Andrew Zisserman
8. Alan L. Yuille

In Geology

1. Or Litany
2. Armand Joulin
3. Ishan Misra
4. Rohit Girdhar
5. Jie Zhou
6. Jiwen Lu
7. Hongsheng Li
8. Yongming Rao

Citations in the corpus

• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, background): “DepthContrast [63] further extends the contrast for depth maps of different augmentations.”
• Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, background): “PointContrast [51] demonstrated the effectiveness of unsupervised contrastive pretraining for 3D scene understanding tasks, with various methods introducing augmentation alternatives for 3D pre-training [45,23,7,57].”
• Unsupervised Point Cloud Pre-training via Occlusion Completion (2020, unknown): “These have been recently outperformed by self-supervised objectives [42, 56, 2, 44, 20, 61].”
• Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement (2022, methodology): “To reduce the labor of human annotation, weakly-supervised methods [21, 22, 29, 33], semi-supervised methods [41, 49] and unsupervised pretraining methods [12,30,45,47] have been proposed recently.”
• Masked Autoencoders for Point Cloud Self-supervised Learning (2022, background): “SSL for Point Cloud SSL has also been widely studied for point cloud representation learning [41,58,48,36,1,20,52,34,13,51].”
• CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (2022, methodology): “In particular, our model outperforms DepthContrast [73], which also employ a cross-modal setting for point cloud representation learning, by a significant margin of 5.”
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, unknown): “A contrastive learning framework is proposed by DepthContrast [64] to learn representations from depth scans.”
• An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “We use the RandomCuboid augmentation from [88] which reduces overfitting.”
• One Million Scenes for Autonomous Driving: ONCE Dataset (2021, background): “For self-supervised learning, PointContrast [47] 116 and DepthContrast [55] apply contrastive learning on point clouds.”