PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers

By Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, Jie Zhou. 2021

632 citations recorded by the source corpus.

Point clouds captured in real-world applications are of-ten incomplete due to the limited sensor resolution, single viewpoint, and occlusion. Therefore, recovering the complete point clouds from partial ones becomes an indispensable task in many practical applications. In this paper, we present a new method that reformulates point cloud completion as a set-to-set translation problem and design a new model, called PoinTr that adopts a transformer encoder-decoder architecture for point cloud completion. By rep-resenting the point cloud as a set of unordered groups of points with position embeddings, we convert the point cloud to a sequence of point proxies and employ the transformers for point cloud generation. To facilitate transformers to better leverage the inductive bias about 3D geometric structures of point clouds, we further devise a geometry-aware block that models the local geometric relationships explicitly. The migration of transformers enables our model to better learn structural knowledge and preserve detailed information for point cloud completion. Furthermore, we propose two more challenging benchmarks with more diverse incomplete point clouds that can better reflect the real-world scenarios to promote future research. Experimental results show that our method outperforms state-of-the-art methods by a large margin on both the new bench-marks and the existing ones. Code is available at https://github.com/yuxumin/PoinTr.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Geology, Model Architectures, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Daniel Cohen-Or
2. Hui Huang
3. Niloy J. Mitra
4. Yunpeng Zhang
5. Justin Johnson
6. Hongsheng Li
7. Zhengbiao Zhu
8. Georgia Gkioxari

In Computer Graphics and Computer-Aided Design

1. Daniel Cohen-Or
2. Niloy J. Mitra
3. Hui Huang
4. Justin Johnson
5. Georgia Gkioxari
6. Jitendra Malik
7. Yu Qiao
8. Zhengbiao Zhu

In Geology

1. Jie Zhou
2. Yunpeng Zhang
3. Jiwen Lu
4. Daniel Cohen-Or
5. Jitendra Malik
6. Yu Qiao
7. Peng Gao
8. Lulu Tang

In Model Architectures

1. Christoph Feichtenhofer
2. Hongsheng Li
3. Jitendra Malik
4. Yu Qiao
5. Peng Gao
6. Justin Johnson
7. Zheng Zhu
8. Dalong Du

In 3D Vision & Pose

1. Georgia Gkioxari
2. Justin Johnson
3. Jitendra Malik
4. Daniel Cohen-Or
5. Hui Huang
6. Christoph Feichtenhofer
7. Chaozheng Wu
8. Yu Qiao

Citations in the corpus

• Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Finally, we compare MCC and its queriable 3D decoder with a state-of-the-art 3D point completion method PoinTr [79].”
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification (2021, background): “Recent progress has demonstrated the variants of transformers can also be a competitive alternative to CNNs and achieve promising results on different vision tasks including image classification [8, 25, 20, 35, 23], object detection [2], semantic segmentation [34, 5] and 3D analysis [31, 33].”
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, methodology): “Some preliminary explorations have been implemented [11,59,63].”
• AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022, neutral): “Ours MPC PoinTr Our approach yields more diverse generations, while also better preserving the originally observed structure.”
• A Survey of Visual Transformers (2021, neutral): “To this end, PoinTr [123] represents the original point cloud as a set of local point proxies and leverages a geometry-aware encoder–decoder Transformer to migrate the center point proxies toward incomplete points direction.”
• Global Filter Networks for Image Classification (2021, background): “The transformer architecture, originally designed for the natural language processing (NLP) tasks [42], has shown promising performance on various vision problems recently [10, 40, 27, 49, 4, 47, 35, 5].”
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, methodology): “PoinTr [58] and Point-BERT [59] divide point clouds into multiple spatial local patches and utilize standard transformers of plain architectures to encode the patches.”
• ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “Also, we only compare state-of-the-art deterministic methods: ConvONet [10], IF-Net [2], and PoinTr [14] in these examples.”
• OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Inspired by the widespread success of vision transformers [13, 34] in various vision tasks [5, 59, 16, 54, 34, 29], we are motivated to utilize the attention mechanism for building the encoder-decoder network for 3D semantic occupancy prediction.”