Masked Autoencoders for Point Cloud Self-supervised Learning

By Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei David Liu, Yonghong Tian, Liuliang Yuan. European Conference on Computer Vision, 2022

737 citations recorded by the source corpus.

As a promising scheme of self-supervised learning, masked autoencoding has significantly advanced natural language processing and computer vision. Inspired by this, we propose a neat scheme of masked autoencoders for point cloud self-supervised learning, addressing the challenges posed by point cloud's properties, including leakage of location information and uneven information density. Concretely, we divide the input point cloud into irregular point patches and randomly mask them at a high ratio. Then, a standard Transformer based autoencoder, with an asymmetric design and a shifting mask tokens operation, learns high-level latent features from unmasked point patches, aiming to reconstruct the masked point patches. Extensive experiments show that our approach is efficient during pre-training and generalizes well on various downstream tasks. Specifically, our pre-trained models achieve 85.18% accuracy on ScanObjectNN and 94.04% accuracy on ModelNet40, outperforming all the other self-supervised learning methods. We show with our scheme, a simple architecture entirely based on standard Transformers can surpass dedicated Transformer models from supervised learning. Our approach also advances state-of-the-art accuracies by 1.5%-2.3% in the few-shot object classification. Furthermore, our work inspires the feasibility of applying unified architectures from languages and images to the point cloud.

View Paper: https://www.semanticscholar.org/paper/a601e71127d95bdae30fd818d2a0cc34b80b13f7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Renrui Zhang
2. Peng Gao
3. Jiaming Han
4. Hongsheng Li
5. Ziyu Guo
6. Yu Qiao
7. Zhijian Liu
8. Chaoning Zhang

Citations in the corpus

• Transformer-Based Visual Segmentation: A Survey (2023, unknown): “OneFormer [271] performs the best on ADE20 K.”
• Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Therein, Point-MAE [50] and Point-M2AE [89] introduce masked autoencoders [26] into 3D point clouds pre-training, achieving competitive results on different 3D tasks.”
• A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “Moreover, the masked tokens from their inputs are processed as the input of Transformers, causing high compute and early leakage of location information [103].”
• A Survey of Visual Transformers (2021, methodology): “Specifically, Point-BERT [113] and Point-MAE [114] directly transfer the previous works [70], [71] to point clouds, while MaskPoint [115] changes the generative training scheme by using a contrastive decoder as similar as DINO (2022) [92] for binary noise/part classification.”
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, background): “We pre-train and fine-tune Point-MAE for 3D object detection both on ScanNetV2 [10].”
• Point Transformer V3: Simpler, Faster, Stronger (2023, background): “While major efforts in 3D representation learning focused on individual objects [57, 68, 69, 87, 103], some recent advancements have redirected attention towards training on real-world scene-centric point clouds [30, 36, 91, 94, 107].”
• PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, methodology): “To address the issues, there has been rapid progress in deep learning-based methods [38, 33, 44, 67], consistently pushing the performance to the new record.”