Spherical CNNs

By Taco Cohen, Mario Geiger, Jonas Köhler, Max Welling. 2018

1,032 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) have become the method of choice for learning problems involving 2D planar images. However, a number of problems of recent interest have created a demand for models that can analyze spherical images. Examples include omnidirectional vision for drones, robots, and autonomous cars, molecular regression problems, and global weather and climate modelling. A naive application of convolutional networks to a planar projection of the spherical signal is destined to fail, because the space-varying distortions introduced by such a projection will make translational weight sharing ineffective. In this paper we introduce the building blocks for constructing spherical CNNs. We propose a definition for the spherical cross-correlation that is both expressive and rotation-equivariant. The spherical correlation satisfies a generalized Fourier theorem, which allows us to compute it efficiently using a generalized (non-commutative) Fast Fourier Transform (FFT) algorithm. We demonstrate the computational efficiency, numerical accuracy, and effectiveness of spherical CNNs applied to 3D model recognition and atomization energy regression.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Directional Message Passing for Molecular Graphs (2020, methodology): “Following work has generalized this principle to spheres (Cohen et al., 2018), molecules (Thomas et al., 2018), volumetric data (Weiler et al., 2018), and general manifolds (Cohen et al., 2019).”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “…et al., 2018; Hoogeboom et al., 2018; Bekkers et al., 2018; Veeling et al., 2018; Lenssen et al., 2018; Graham et al., 2020) to spherical rotations (Cohen et al., 2018; Worrall & Brostow, 2018; Weiler et al., 2018a; Cohen et al., 2019b), scaling (Marcos et al., 2018; Worrall & Welling, 2019;…”
SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020, methodology): “Our method sets itself apart by using roto-translation equivariant layers acting directly on the point cloud without prior projection onto a sphere [22, 45, 7].”
Isometric Transformation Invariant and Equivariant Graph Convolutional Networks (2020, background): “Subsequent research works generalized it to continuous groups (Cohen et al., 2018), three-dimensional data (Weiler et al., 2018), and general manifolds (Cohen et al., 2019).”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, methodology): “While the approach has been applied on general data types like spherical images (Esteves et al., 2018; Cohen et al., 2018; Jiang et al., 2019), voxel data (Weiler et al., 2018), and point clouds (Thomas et al., 2018; Anderson et al., 2019), the requirement of working out the representation theory…”
Set2Graph: Learning Graphs From Sets (2020, background): “…images with rotational and reﬂectional symmetries (Cohen & Welling, 2016a;b; Dieleman et al., 2016; Worrall et al., 2017), learning functions on spheres (Cohen et al., 2018; Esteves et al., 2017) and learning general 3D data (Weiler et al., 2018; Worrall & Brostow, 2018; Weiler et al., 2018).”
General E(2)-Equivariant Steerable CNNs (2019, methodology): “Our results can therefore readily be transferred to e.g. spherical CNNs [19, 5, 20, 21, 22, 23] or more general models of geometric deep learning [24, 25, 26, 27].”
Cormorant: Covariant Molecular Neural Networks (2019, methodology): “This is the approach taken in e.g., [Cohen et al., 2018].”
Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds (2018, background): “Cohen et al. [8] use spherical harmonics and Wigner D -matrices but only address spherical signals (2D data on the surface of a sphere).”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, neutral): “This approach works for any homogeneous space , where by deﬁnition it is possible to move from any point p ∈ M to any other point q ∈ M using an appropriate symmetry transformation (Kondor & Trivedi, 2018; Cohen et al., 2018c;a).”
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, background): “A recent major success of this approach are Spherical CNNs [1][8], which are an SO(3)–equivariant neural network architecture for learning images painted on the sphere3.”
RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks (2018, neutral): “SO (3)-equivariant CNN for signals on spheres was studied in [4] in a diﬀerent setting.”
Learning to Reconstruct Shapes from Unseen Classes (2018, methodology): “As spherical maps are signals on the unit sphere, it is tempting to use network architectures based on spherical convolution [Cohen et al., 2018].”
On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups (2018, methodology): “In particular, we argue for Fourier space representations, similar to those that have appeared in (Worrall et al., 2017; Cohen et al., 2018; Kondor et al., 2018)), and, even more recently, since the submission of the original version of the present paper in (Thomas et al., 2018; Kondor, 2018;…”
N-body Networks: a Covariant Hierarchical Neural Network Architecture for Learning Atomic Potentials (2018, background): “…al., 2016; Gilmer et al., 2017; Kondor et al., 2018), manifolds (Monti et al., 2016; Masci et al., 2015), and other domains (Bruna & Mallat, 2013; Cohen et al., 2018), as well as the question of generalizing the concept of equivariance (covariance) in general (Cohen & Welling, 2016; 2017; Kondor…”
A General Theory of Equivariant CNNs on Homogeneous Spaces (2018, methodology): “In [10], the authors choose a spectral approach to represent the signal and kernels in Fourier space.”
Spherical CNNs on Unstructured Grids (2019, methodology): “We benchmark our results against PointNet++ (Qi et al., 2017a), VoxNet (Qi et al., 2016), and S2CNN 2 .”
CubeNet: Equivariance to 3D Rotation and Translation (2018, background): “However, with the exception of Cohen and Welling [9] (projections on sphere), Kondor [28] (point clouds), and Thomas et al.”