PU-Net: Point Cloud Upsampling Network

By Lequan Yu, Chi-Wing Fu, Daniel Cohen-Or, P. Heng. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

730 citations recorded by the source corpus.

Learning and analyzing 3D point clouds with deep networks is challenging due to the sparseness and irregularity of the data. In this paper, we present a data-driven point cloud upsampling technique. The key idea is to learn multi-level features per point and expand the point set via a multi-branch convolution unit implicitly in feature space. The expanded feature is then split to a multitude of features, which are then reconstructed to an upsampled point set. Our network is applied at a patch-level, with a joint loss function that encourages the upsampled points to remain on the underlying surface with a uniform distribution. We conduct various experiments using synthesis and scan data to evaluate our method and demonstrate its superiority over some baseline methods and an optimization-based method. Results show that our upsampled points have better uniformity and are located closer to the underlying surfaces.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Geology

View Paper: https://www.semanticscholar.org/paper/c82ca047a80dbdf10ec6a92f860c0c40035c6517

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Geology

Citations in the corpus

Unsupervised 3D Shape Completion through GAN Inversion (2021, methodology): “We compare our PatchVariance against expansion penalty [21] and repulsion loss [35].”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “Each Gaussian is characterized by a covariance matrix Σ and a center point X , which is referred to as the mean value of the Gaussian: For differentiable optimization, the covariance matrix Σ can be decomposed into a scaling matrix S and a rotation matrix R : the computation of gradient flow during…”
Pointwise Convolutional Neural Networks (2017, background): “In addition, it would be of great interest to extend pointwise convolutional neural networks to geometry point cloud processing [44], or explore the connection of pointwise convolution to tensor voting [41], which was used in the literature to detect structures in a local point neighborhood.”
PF-Net: Point Fractal Network for 3D Point Cloud Completion (2020, positive): “Thus, algorithms which can directly conduct shape completion task on unordered point have been significantly developed [19–23].”
EC-Net: an Edge-aware Point set Consolidation Network (2018, methodology): “Inspired by PointNet [9], we directly process 3D points by converting their coordinates into deep features and producing more points by feature expansion [7].”
Patch-Based Progressive 3D Point Set Upsampling (2018, methodology): “[59] introduced the first end-to-end point set upsampling network, PU-Net, where both the input and the output are the 3D coordinates of a point set.”
Monte Carlo convolution for learning on non-uniformly sampled point clouds (2018, background): “Follow up work has proposed several extensions [Atzmon et al. 2018; Graham and van der Maaten 2017; Groh et al. 2018; Huang et al. 2018; Li et al. 2018; Shen et al. 2017; Su et al. 2018; Wang et al. 2017, 2018; Xu et al. 2018; Yu et al. 2018].”
PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling (2020, methodology): “Furthermore, in order to avoid the sampled points being too close to each other in some local areas after the AS module transformation, we also use Repulsion Loss [16] to restrict the deformation of sampled point clouds.”
Point-Based Multi-View Stereo Network (2019, methodology): “Therefore, we compare our method with PU-Net [30], where multi-level features are extracted from the coarse point cloud to reconstruct an upsampled point cloud.”
PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors (2018, background): “FoldingNet [48] While PointNet can work with point clouds, it is still a supervised architecture, and constructing unsupervised extensions like an auto-encoder on points is non-trivial as the upsampling step is required to interpolate sets [50, 31].”
3D Point Capsule Networks (2018, methodology): “PU-Net [41] proposed better upsampling schemes to be used in decoding.”
ABC: A Big CAD Model Dataset for Geometric Deep Learning (2018, methodology): “Since the ground truth geometry is known for B-rep models, they can be used to simulate a scanning setup and quantitatively evaluate the reconstruction errors of both reconstruction [15, 56] and point cloud upsampling [64] techniques.”
JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds (2019, background): “2016; Maturana and Scherer 2015) and 3D point clouds (Qi et al. 2017b; Li et al. 2018; Wang et al. 2018b; Yu et al. 2018).”
PU-GAN: A Point Cloud Upsampling Adversarial Network (2019, methodology): “To upsample a point set, PU-Net [40] duplicates the point features and uses separate MLPs to process each copy independently.”
Multimodal Virtual Point 3D Detection (2021, methodology): “Lidar-based methods like PUNet [69], PUGAN [27], and Wang et al. [64] learn high level point-wise features from raw Lidar scans.”
SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (2021, result): “Compared with the previous methods, which often ignore the spatial relationship among points [56, 43, 31] or simply learn through self-attention in a single level of multi-step point cloud decoding [53, 29, 46], our skip-transformer is proposed to integrate the spatial relationships across different”
PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows (2019, background): “Recently, substantial progress has been made in point cloud synthesis tasks such as auto-encoding [1, 51, 17], singleview 3D reconstruction [12, 22, 29, 31, 13], stereo reconstruction [45], and point cloud completion [54, 53] Many point cloud synthesis works convert a point distribution to a N × 3 m”