Learning SO(3) Equivariant Representations with Spherical CNNs

By Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, Kostas Daniilidis. 2017

582 citations recorded by the source corpus.

We address the problem of 3D rotation equivariance in convolutional neural networks. 3D rotations have been a challenging nuisance in 3D classification tasks requiring higher capacity and extended data augmentation in order to tackle it. We model 3D data with multi-valued spherical functions and we propose a novel spherical convolutional network that implements exact convolutions on the sphere by realizing them in the spherical harmonic domain. Resulting filters have local symmetry and are localized by enforcing smooth spectra. We apply a novel pooling on the spectral domain and our operations are independent of the underlying spherical resolution throughout the network. We show that networks with much lower capacity and without requiring data augmentation can exhibit performance comparable to the state of the art in standard 3D shape retrieval and classification benchmarks.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Perceiver: General Perception with Iterative Attention (2021, background): “, 2019), or for groups other than the group of linear translations (e.g. Cohen & Welling 2016; Bronstein et al. 2017; Esteves et al. 2018)); this however often requires a tremendous amount of researcher time and expertise.”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, methodology): “While the approach has been applied on general data types like spherical images (Esteves et al., 2018; Cohen et al., 2018; Jiang et al., 2019), voxel data (Weiler et al., 2018), and point clouds (Thomas et al., 2018; Anderson et al., 2019), the requirement of working out the representation theory…”
General E(2)-Equivariant Steerable CNNs (2019, neutral): “Our results can therefore readily be transferred to e.g. spherical CNNs [19, 5, 20, 21, 22, 23] or more general models of geometric deep learning [24, 25, 26, 27].”
Cormorant: Covariant Molecular Neural Networks (2019, methodology): “…of using “Fourier space” activations, i.e., expressing the activations of neurons in a basis deﬁned by the irreducible representations of the underlying symmetry group (see also [Esteves et al., 2017]), and these ideas were later generalized to the entire SE (3) group [Weiler et al., 2018].”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, methodology): “Spherical CNNs Besides the general theoretical framework of gauge equivariant convolution, we present in this paper a speciﬁc model (the Icosahedral CNN), which can be viewed as a fast and simple alternative to Spherical CNNs (Cohen et al., 2018b; Esteves et al., 2018; Boomsma & Frellsen, 2017; Su &”
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, background): “A recent major success of this approach are Spherical CNNs [1][8], which are an SO(3)–equivariant neural network architecture for learning images painted on the sphere3.”
Learning to Reconstruct Shapes from Unseen Classes (2018, methodology): “Spherical projections have been shown effective in 3D shape retrieval [Esteves et al., 2018], classification [Cao et al.”
Spherical CNNs on Unstructured Grids (2019, background): “Very recent work, such as Cohen et al. (2018) and Esteves et al. (2018), propose network architectures that operate natively in the spherical domain, and are invariant to rotations in the SO (3) group.”
Deep Scale-spaces: Equivariance Over Scale (2019, unknown): “…2016a, Weiler et al., 2018b, Bekkers et al., 2018, Hoogeboom et al., 2018] and 3D [Worrall et al., 2017], continuous rotations on the sphere [Esteves et al., 2018, Cohen et al., 2018b], in-plane reﬂections [Cohen and Welling, 2016a], and even reverse-conjugate symmetry in DNA sequencing…”
The Perfect Match: 3D Point Cloud Matching With Smoothed Densities (2018, background): “However, [5, 7] find that this strategy often fails for 3D point clouds.”
Learning to Convolve: A Generalized Weight-Tying Approach (2019, background): “…et al., 2018) and discrete roto-translations in 2D (Cohen & Welling, 2016a; Weiler et al., 2018b; Bekkers et al., 2018; Hoogeboom et al., 2018) and 3D (Worrall et al., 2017), continuous rotations on the sphere (Esteves et al., 2018; Cohen et al., 2018), in-plane reﬂections (Cohen & Welling, 2016a).”
Kernel Transformer Networks for Compact Spherical Convolution (2018, methodology): “This is possible because the KTN takes the source CNN as input rather than embed the CNN kernels into its own parameters (unlike [11,12,15,36,45]).”
Vector Neurons: A General Framework for SO(3)-Equivariant Networks (2021, methodology): “In classiﬁcation and segmentation, following the conventions from Esteves et al. [11], we adopt three train / test settings: z/z , z/ SO(3) and SO(3) / SO(3) , where z stands for data augmentation with rotations only around the z axis, and SO(3) for arbitrary rotations.”
GemNet: Universal Directional Graph Neural Networks for Molecules (2021, background): “Equivariant models for the SO(3) group were ﬁrst investigated in the context of spherical convolutions by Cohen et al. [14], Esteves et al. [21], Kondor et al. [36].”
DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications (2018, methodology): “Ideally, we would compare DeepSphere to alternative spherical CNNs like [24] and [25].”
B-Spline CNNs on Lie Groups (2019, methodology): “Related work by Tai et al. (2019) relies on the same Lie group principles as we do in this paper (the Log map) to construct convenient coordinate systems, such as log-polar coordinates Esteves et al. (2018b), to handle equivariance.”
Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution (2018, methodology): “Type II: Azimuthally isotropic convolution: This spherical convolution outputs a function defined on S2 using an azimuthally symmetric filter h0(û) (Esteves et al. (2018); Driscoll & Healy (1994)): h0(û) , ∫ π 0 1 2π (DRz(ω)h)(û)dω (8) h} f(R) = g(ϕ, ϑ) , ∫ S2 K∑ k=0 (DRz(ϕ) ◦…”