SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation

By Li Yi, Hao Su, Xingwen Guo, Leonidas J. Guibas. 2016

491 citations recorded by the source corpus.

In this paper, we study the problem of semantic annotation on 3D models that are represented as shape graphs. A functional view is taken to represent localized information on graphs, so that annotations such as part segment or keypoint are nothing but 0-1 indicator vertex functions. Compared with images that are 2D grids, shape graphs are irregular and non-isomorphic data structures. To enable the prediction of vertex functions on them by convolutional neural networks, we resort to spectral CNN method that enables weight sharing by parametrizing kernels in the spectral domain spanned by graph Laplacian eigenbases. Under this setting, our network, named SyncSpecCNN, strives to overcome two key challenges: how to share coefficients and conduct multi-scale analysis in different parts of the graph for a single shape, and how to share information across related but different shapes that may be represented by very different graphs. Towards these goals, we introduce a spectral parametrization of dilated convolutional kernels and a spectral transformer network. Experimentally we tested SyncSpecCNN on various tasks, including 3D shape part segmentation and keypoint prediction. State-of-the-art performance has been achieved on all benchmark datasets.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017, background): “In 3D metric space, other than point set, there are several popular representations for deep learning, including volumetric grids [21; 22; 29], and geometric graphs [3; 15; 33].”
RealPoint3D: Point Cloud Generation from a Single Image with Complex Background (2018, methodology): “In the encoder part, the 2D Convolutional neural network (CNN) is used to extract the features from 2D image.”
Geometric Deep Learning: Going beyond Euclidean data (2016, methodology): “A possible remedy to the generalization problem of spectral methods is the recent architecture proposed in [118], applying the idea of spatial transformer networks [119] in the spectral”
Generating 3D faces using Convolutional Mesh Autoencoders (2018, background): “[48] also present a spectral CNN for labeling nodes but does not involve any mesh dimensionality reduction.”
Dynamic Graph CNN for Learning on Point Clouds (2018, background): “ulty, however, is that the Laplacian eigenbasis is domain-dependent; thus, a ﬁlter learned on one shape may not generalize to others. Spectral transformer networks address this problem to some extent [56]. An alternative deﬁnition of non-Euclidean convolution employs spatial rather than spectral ﬁlt”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, background): “[38] adapt the idea of graph convolutions in the spectral domain to semantic segmentation of 3D shapes.”
3D shape segmentation via shape fully convolutional networks (2017, neutral): “These methods [30, 31] proposed different deep archi8”
Learning SO(3) Equivariant Representations with Spherical CNNs (2017, background): “There are two threads within this space, spectral (Bruna et al. 2013b; Defferrard et al. 2016; Kipf and Welling 2016; Yi et al. 2016) and spatial (Boscaini et al.”
Neural 3D Mesh Renderer (2017, background): “Although recognition and segmentation have been investigated [10, 38], generative tasks are much more difficult.”
Spherical CNNs on Unstructured Grids (2019, methodology): “Other methods perform graph convolution by parameterizing the convolution kernels in the spectral domain, thus converting the convolution step into a spectral dot product (Bruna et al., 2014; Defferrard et al., 2016; Kipf & Welling, 2017; Yi et al., 2017).”
PointCNN: Convolution On X-Transformed Points (2018, background): “ShapeNet Parts S3DIS ScanNet pIoU mpIoU mIoU OA SyncSpecCNN [55] 84.74 82.0 - - Pd-Network [22] 85.49 82.7 - - SSCN [12] 85.98 83.3 - - SPLATNet [43] 85.4 83.7 - - SpiderCNN [53] 85.3 81.7 - - SO-Net [27] 84.9 81.0 - - PCNN [3] 85.1 81.8 - - KCNet [42] 83.7 82.2 - - SpecGCN [46] 85.4 - - - Kd-Net [2”
Rotation Invariant Convolutions for 3D Point Clouds Deep Learning (2019, methodology): “Some methods exploit hierarchical structures and clustering for scalability [28, 15, 33, 34, 16], mapping point cloud to two dimensional space [40, 9, 18], applying spectral analysis [42], or addressing non-uniform point distribution [10].”
Local Spectral Graph Convolution for Point Set Feature Learning (2018, methodology): “We use the ofﬁcial training/testing split, following [11,20,21], where the challenge is to assign a part label to each point in the test set.”
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds (2017, background): “Yi et al. [31] use spectral CNNs on 3D models represented as shape graphs for shape part segmentation.”
Relation-Shape Convolutional Neural Network for Point Cloud Analysis (2019, unknown): “6 SyncCNN [50] mesh 82.”
Convolutional neural networks on surfaces via seamless toric covers (2017, methodology): “Another disadvantage of spectral methods (which is targeted by [Yi et al. 2016]) is their difficulty with cross-shape learning which stems from the fact that the spectral decomposition of each shape can be inconsistent.”
PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding (2020, background): “g unordered sets [47, 50, 80, 39]. Second, is the choice of neighborhood aggregation mechanism which could either be hierarchical [48, 33, 82, 16, 35], spatial CNN-like [30, 73, 37, 85, 59], spectral [78, 62, 65] or graph-based [72, 64, 68, 54]. Finally, since the points are discrete samples of an u”