CubeNet: Equivariance to 3D Rotation and Translation

By Daniel E. Worrall, Gabriel J. Brostow. 2018

152 citations recorded by the source corpus.

3D Convolutional Neural Networks are sensitive to transformations applied to their input. This is a problem because a voxelized version of a 3D object, and its rotated clone, will look unrelated to each other after passing through to the last layer of a network. Instead, an idealized model would preserve a meaningful representation of the voxelized object, while explaining the pose-difference between the two inputs. An equivariant representation vector has two components: the invariant identity part, and a discernable encoding of the transformation. Models that can't explain pose-differences risk "diluting" the representation, in pursuit of optimizing a classification or regression loss function. We introduce a Group Convolutional Neural Network with linear equivariance to translations and right angle rotations in three dimensions. We call this network CubeNet, reflecting its cube-like symmetry. By construction, this network helps preserve a 3D shape's global and local signature, as it is transformed through successive layers. We apply this network to a variety of 3D inference problems, achieving state-of-the-art on the ModelNet10 classification challenge, and comparable performance on the ISBI 2012 Connectome Segmentation Benchmark. To the best of our knowledge, this is the first 3D rotation equivariant CNN for voxel representations.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “…et al., 2018; Bekkers et al., 2018; Veeling et al., 2018; Lenssen et al., 2018; Graham et al., 2020) to spherical rotations (Cohen et al., 2018; Worrall & Brostow, 2018; Weiler et al., 2018a; Cohen et al., 2019b), scaling (Marcos et al., 2018; Worrall & Welling, 2019; Sosnovik et al., 2020;…”
Set2Graph: Learning Graphs From Sets (2020, background): “…images with rotational and reﬂectional symmetries (Cohen & Welling, 2016a;b; Dieleman et al., 2016; Worrall et al., 2017), learning functions on spheres (Cohen et al., 2018; Esteves et al., 2017) and learning general 3D data (Weiler et al., 2018; Worrall & Brostow, 2018; Weiler et al., 2018).”
General E(2)-Equivariant Steerable CNNs (2019, methodology): “Via a group pooling or projection operation max : R c → R , f ( x ) → max( f ( x )) the works [6, 7, 9, 32, 31] extract the maximum value of a regular or quotient ﬁeld.”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, background): “…rotations of the plane and 3D space (Oyallon & Mallat, 2015; Cohen & Welling, 2016; 2017; Marcos et al., 2017; Weiler et al., 2018b;a; Worrall et al., 2017; Worrall & Brostow, 2018; Winkels & Cohen, 2018; Winkens et al., 2018; Thomas et al., 2018; Bekkers et al., 2018), and the sphere (see below).”
A General Theory of Equivariant CNNs on Homogeneous Spaces (2018, background): “Some other examples include an RGB image (ρ(h) = I3), a field of wind directions on earth (ρ(h) a 2×2 rotation matrix), a diffusion tensor MRI image (ρ(h) a representation of SO(3) acting on 2-tensors), a regular G-CNN on Z(3) [14, 15] (ρ a regular representation of H).”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, methodology): “Although there are significant improvements in 3D DL models, such as PointNet [10], PointNet++ [12], PointCNN [71], DGCNN [74], RotationNet [78] and other work [52, 139–141].”
Wavelet Networks: Scale Equivariant Learning From Raw Waveforms (2020, methodology): “[77] Daniel Worrall and Gabriel Brostow.”
B-Spline CNNs on Lie Groups (2019, background): “It can be roughly divided into work on discrete G-CNNs (Cohen & Welling, 2016; Dieleman et al., 2016; Winkels & Cohen, 2018; Worrall & Brostow, 2018; Hoogeboom et al., 2018), regular continuous G-CNNs (Oyallon & Mallat, 2015; Bekkers et al., 2015a; 2018b; Weiler et al., 2018a; Zhou et al., 2017;…”
Geometric and Physical Quantities improve E(3) Equivariant Message Passing (2021, background): “Alternatively, the kernels are only transformed via a sub-group of transformations in E( 3 ) that leaves the grid on which the kernel is deﬁned intact, as in (Worrall & Brostow, 2018; Winkels & Cohen, 2018).”
Attentive Group Equivariant Convolutional Networks (2020, background): “…2018; Li et al., 2018; Cheng et al., 2018; Bekkers et al., 2018; Lenssen et al., 2018; Smets et al., 2020), spherical rotations (Cohen et al., 2018; Worrall & Brostow, 2018; Cohen et al., 2019b; Thomas et al., 2018; Kondor & Trivedi, 2018), scaling (Marcos et al., 2018; Worrall & Welling, 2019;…”
On Learning Sets of Symmetric Elements (2020, unknown): “…rotations (Dieleman et al., 2016; Cohen & Welling, 2016a;b; Worrall et al., 2017), spherical symmetries (Cohen et al., 2018; 2019b; Esteves et al., 2017), or 3D symmetries (Weiler et al., 2018; Winkels & Cohen, 2018; Worrall & Brostow, 2018; Kondor, 2018; Thomas et al., 2018; Weiler et al., 2018).”
Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring In Data (2019, background): “…e.g., planar rotations (Dieleman et al., 2016; Marcos et al., 2017; Weiler et al., 2018; Li et al., 2018), spherical rotations (Cohen et al., 2018; Worrall & Brostow, 2018; Cohen et al., 2019), scaling (Marcos et al., 2018; Worrall & Welling, 2019) and general transformation groups (Cohen &…”