3D Shape Segmentation with Projective Convolutional Networks

By Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, Siddhartha Chaudhuri. 2016

381 citations recorded by the source corpus.

This paper introduces a deep architecture for segmenting 3D objects into their labeled semantic parts. Our architecture combines image-based Fully Convolutional Networks (FCNs) and surface-based Conditional Random Fields (CRFs) to yield coherent segmentations of 3D shapes. The image-based FCNs are used for efficient view-based reasoning about 3D object parts. Through a special projection layer, FCN outputs are effectively aggregated across multiple views and scales, then are projected onto the 3D object surfaces. Finally, a surface-based CRF combines the projected outputs with geometric consistency cues to yield coherent segmentations. The whole architecture (multi-view FCNs and CRF) is trained end-to-end. Our approach significantly outperforms the existing state-of-the-art methods in the currently largest segmentation benchmark (ShapeNet). Finally, we demonstrate promising segmentation results on noisy 3D shapes acquired from consumer-grade depth cameras.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

3D shape segmentation via shape fully convolutional networks (2017, methodology): “[18, 29] applied deep architectures to produce the mesh 26”
Neural 3D Mesh Renderer (2017, background): “Although recognition and segmentation have been investigated [10, 38], generative tasks are much more difficult.”
PointGrid: A Deep Network for 3D Shape Understanding (2018, unknown): “[22] proposed a more complete multi-view framework.”
Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline (2021, methodology): “…point-cloud analysis problems like segmentation (Ladick ` y et al., 2010; Tighe & Lazeb-nik, 2010; Riemenschneider et al., 2014; Qin et al., 2018; Dai & Nießner, 2018; Kalogerakis et al., 2017; Tatarchenko et al., 2018), reconstruction (Pittaluga et al., 2019) and rendering (Aliev et al., 2019).”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, background): “Very recently, a multi-view network has been proposed for part-based mesh segmentation [26].”
Multi-directional geodesic neural networks via equivariant convolution (2018, methodology): “2D images, for example by rendering multiple views of the object. Some of the earliest variants of this idea include methods that represent shapes as unordered collections of views (or range images) [Kalogerakis et al. 2017; Su et al. 2015; Wei et al. 2016] or exploit the panorama image representati”
SPLATNet: Sparse Lattice Networks for Point Cloud Processing (2018, background): “Multi-view networks pre-process shapes into a set of 2D rendered images encoding surface depth and normals under various 2D projections [39, 32, 3, 24, 9, 20].”
Tangent Convolutions for Dense Prediction in 3D (2018, methodology): “In a more controlled setting with fixed camera poses, multi-view methods are successfully used for shape segmentation [21], shape recognition [49, 40], and shape synthesis [14, 30, 22, 50].”
MVTN: Multi-View Transformation Network for 3D Shape Recognition (2020, background): “pressive performance, and in many instances, achieve stateof-the-art results in 3D shape classification and segmentation [40, 75, 45, 39, 16].”
DiffusionNet: Discretization Agnostic Learning on Surfaces (2020, methodology): “…methods directly leveraged tools developed for 2D images, and thus mapped 3D shapes onto the plane either using multi-view renderings [Kalogerakis et al. 2017; Su et al. 2015; Wei et al. 2016] or more global, often parametrization-based techniques such as panoramas [Sfikas et al.…”
Multiresolution Tree Networks for 3D Point Cloud Processing (2018, methodology): “Multiview methods [39,31,37,28,20] represent a 3D shape as images rendered from different viewpoints.”
Learning Representations and Generative Models for 3D Point Clouds (2017, background): “Recently, deep learning architectures for view-based projections (Su et al., 2015; Wei et al., 2016; Kalogerakis et al., 2016), volumetric grids (Qi et al.”
Deep Learning for 3D Point Clouds: A Survey (2019, background): “[249] combined FCNs and surface-based CRFs to achieve end-to-end 3D part segmentation.”
On Learning Sets of Symmetric Elements (2020, background): “Learning with data that consists of unordered sets of elements is an important problem with numerous applications, from classiﬁcation and segmentation of 3D data (Zaheer et al., 2017; Qi et al., 2017; Su et al., 2015; Kalogerakis et al., 2017) to image deblurring (Aittala & Durand, 2018).”
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans (2018, background): “recently proposed an approach for shape segmentation by projecting predicted 2D confidence maps onto the 3D shape, which are then aggregated through a CRF [14].”
Deep Volumetric Video From Very Sparse Multi-view Performance Capture (2018, background): “Multi-view convolutional neural networks (CNN) have been introduced to learn deep features for various 3D tasks including shape segmentation [57], object recognition and classiﬁcation [58–60], correspondence matching [61], and novel view synthesis [62–64].”
Review: deep learning on 3D point clouds (2020, methodology): “These methods [17, 18, 19, 10, 20, 21, 22], take advantage of the already matured 2D CNNs into 3D.”