Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

By Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, Hongsheng Li. Neural Information Processing Systems, 2022

417 citations recorded by the source corpus.

Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. However, it still remains an open question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds. In this paper, we propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we modify the encoder and decoder into pyramid architectures to progressively model spatial geometries and capture both fine-grained and high-level semantics of 3D shapes. For the encoder that downsamples point tokens by stages, we design a multi-scale masking strategy to generate consistent visible regions across scales, and adopt a local spatial self-attention mechanism during fine-tuning to focus on neighboring patterns. By multi-scale token propagation, the lightweight decoder gradually upsamples point tokens with complementary skip connections from the encoder, which further promotes the reconstruction from a global-to-local perspective. Extensive experiments demonstrate the state-of-the-art performance of Point-M2AE for 3D representation learning. With a frozen encoder after pre-training, Point-M2AE achieves 92.9% accuracy for linear SVM on ModelNet40, even surpassing some fully trained methods. By fine-tuning on downstream tasks, Point-M2AE achieves 86.43% accuracy on ScanObjectNN, +3.36% to the second-best, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme. Code is available at https://github.com/ZrrSkywalker/Point-M2AE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/34103f1294844c447fe8872bf5c3ab1c7ce32103

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Point-M2AE [259] designs a multiscale MIM pretraining by making the encoder and decoder into pyramid architectures to model spatial geometries and multilevel semantics progressively.”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, background): “Pretrained models have shown promising adaptability for various downstream tasks, such as object detection [46], semantic segmentation [9], and 3D recognition [8, 22, 23, 29, 77, 80, 81].”
PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, background): “Several works further introduce GPT scheme [6], multi-scale [61, 60], and multi-modal [11, 42, 43] to guide 3D representation learning.”
Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning (2022, background): “Recently, this multiscale idea has been extended to vision transformers by [18], who show that this architecture improves various video recognition and image classification tasks, as well as in [21, 65] which proposes various hybrid CNN-MAE architectures that yield multiscale features during MAE pre”
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, background): “The accuracy of vision tasks, such as classification [35, 22, 26, 13, 42, 17, 70], detection [51, 5, 72, 64, 69, 9] and 3D understanding [47, 68, 63, 67] has been boosted significantly thanks”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Therein, Point-MAE [50] and Point-M2AE [89] introduce masked autoencoders [26] into 3D point clouds pre-training, achieving competitive results on different 3D tasks.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “Resembling Point-MAE, [105] proposes Point-M2AE, a Multi-scale MAE for point clouds.”