PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding

By Saining Xie, Jiatao Gu, Demi Guo, Charles Ruizhongtai Qi, Leonidas J. Guibas, Or Litany. European Conference on Computer Vision, 2020

838 citations recorded by the source corpus.

Arguably one of the top success stories of deep learning is transfer learning. The finding that pre-training a network on a rich source set (eg., ImageNet) can help boost performance once fine-tuned on a usually much smaller target set, has been instrumental to many applications in language and vision. Yet, very little is known about its usefulness in 3D point cloud understanding. We see this as an opportunity considering the effort required for annotating data in 3D. In this work, we aim at facilitating research on 3D representation learning. Different from previous works, we focus on high-level scene understanding tasks. To this end, we select a suite of diverse datasets and tasks to measure the effect of unsupervised pre-training on a large source set of 3D scenes. Our findings are extremely encouraging: using a unified triplet of architecture, source dataset, and contrastive loss for pre-training, we achieve improvement over recent best results in segmentation and detection across 6 different benchmarks for indoor and outdoor, real and synthetic datasets -- demonstrating that the learned representation can generalize across domains. Furthermore, the improvement was similar to supervised pre-training, suggesting that future efforts should favor scaling data collection over more detailed annotation. We hope these findings will encourage more research on unsupervised pretext task design for 3D deep learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/491188ac198663094537cb4c38ac1c8808ef7440

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts (2020, methodology): “3 PointContrast (PointInfoNCE) [65] 60.”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, background): “PointContrast [46] demonstrated the effectiveness of unsupervised contrastive pretraining for 3D scene understanding tasks, with various methods introducing augmentation alternatives for 3D pre-training [41,20,7,52].”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, methodology): “[105] proposed a self-supervised method to build representations of scene level point clouds.”
Dense Contrastive Learning for Self-Supervised Visual Pre-Training (2020, background): “Contrastive learning, holds the key to most state-of-the-art methods [41, 17, 2, 42], in which the positive pair is usually formed with two augmented views of the same image (or other visual patterns), while negative ones are formed with different images.”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”
Unsupervised Point Cloud Pre-training via Occlusion Completion (2020, background): “These have been recently outperformed by self-supervised objectives [42, 56, 2, 44, 20, 61].”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, background): “From another perspective, PointContrast [55] utilizes contrastive learning between features of the same points from different views to learn discriminative 3D representations.”
One Million Scenes for Autonomous Driving: ONCE Dataset (2021, methodology): “PointContrast defines a contrastive loss over the point-level features given a pair of overlapping partial scans.”
Point Transformer V3: Simpler, Faster, Stronger (2023, methodology): “While major efforts in 3D representation learning focused on individual objects [57, 68, 69, 87, 103], some recent advancements have redirected attention towards training on real-world scene-centric point clouds [30, 36, 91, 94, 107].”
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders (2023, background): “To tackle this issue, our new insight is to view the masked image from a “sparse data perspective”, which was inspired by learning on sparse point clouds in 3D tasks [15,76].”
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (2022, methodology): “PointContrast [59] performs point-level invariant mapping on two transformed view of the given point cloud.”
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, background): “More recently, several studies suggested using SSL techniques for point cloud understanding [10,14,22,30,36,38,39,42,50,54,57].”
Masked Autoencoders for Point Cloud Self-supervised Learning (2022, background): “SSL for Point Cloud SSL has also been widely studied for point cloud representation learning [41,58,48,36,1,20,52,34,13,51].”
Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement (2022, methodology): “To reduce the labor of human annotation, weakly-supervised methods [21, 22, 29, 33], semi-supervised methods [41, 49] and unsupervised pretraining methods [12,30,45,47] have been proposed recently.”