CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding

By Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, Ranga Rodrigo. Computer Vision and Pattern Recognition, 2022

359 citations recorded by the source corpus.

Manual annotation of large-scale point cloud dataset for varying tasks such as 3D object classification, segmentation and detection is often laborious owing to the irregular structure of point clouds. Self-supervised learning, which operates without any human labeling, is a promising approach to address this issue. We observe in the real world that humans are capable of mapping the visual concepts learnt from 2D images to understand the 3D world. Encouraged by this insight, we propose CrossPoint, a simple cross-modal contrastive learning approach to learn transferable 3D point cloud representations. It enables a 3D-2D correspondence of objects by maximizing agreement between point clouds and the corresponding rendered 2D image in the invariant space, while encouraging invariance to transformations in the point cloud modality. Our joint training objective combines the feature correspondences within and across modalities, thus ensembles a rich learning signal from both 3D point cloud and 2D image modalities in a self-supervised fashion. Experimental results show that our approach outperforms the previous unsupervised learning methods on a diverse range of downstream tasks including 3D object classification and segmentation. Further, the ablation studies validate the potency of our approach for a better point cloud understanding. Code and pretrained models are available at https://github.com/MohamedAfham/CrossPoint.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/0fd08c1237f80d96a6618d93cb1292b45b9f09fc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Yu Qiao
2. Rongyao Fang
3. Peng Gao
4. P. Heng
5. Bin Zhao
6. Renrui Zhang
7. Dong Wang
8. Hongsheng Li

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Peng Gao
3. Yu Qiao
4. Renrui Zhang
5. P. Heng
6. Jiaming Han
7. Xiaogang Wang
8. Ziyu Guo

In Geology

1. Yu Qiao
2. Peng Gao
3. P. Heng
4. Renrui Zhang
5. Hongsheng Li
6. Ziyu Guo

Citations in the corpus

• Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “To extend its application scenarios, numerous efforts [1,23, 92,95] have been made to incorporate 3D point clouds with data from other modalities, allowing for improved 3D understanding [1, 23], text-to-3D generation [35, 49, 52], and 3D question answering [3, 28].”
• Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, background): “Although CaFo has unified four types of pre-training, our future direction will focus on integrating more existing pre-trained knowledge, such as the masked-generative MAE [30], the 3D-contrastive CrossPoint [1], and 3D-generative I2P-MAE [79].”
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, background): “After self-supervised pre-training on ShapeNet [7], Point-M2AE achieves 92.9% classification accuracy for linear SVM on ModelNet40 [53] with the frozen encoder, which surpasses the runner-up CrossPoint [2] by +1.2% and even outperforms some fully supervised methods.”