Harmonic Networks: Deep Translation and Rotation Equivariance

By Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, Gabriel J. Brostow. 2016

783 citations recorded by the source corpus.

Translating or rotating an input image should not affect the results of many computer vision tasks. Convolutional neural networks (CNNs) are already translation equivariant: input image translations produce proportionate feature map translations. This is not the case for rotations. Global rotation equivariance is typically sought through data augmentation, but patch-wise equivariance is more difficult. We present Harmonic Networks or H-Nets, a CNN exhibiting equivariance to patch-wise translation and 360-rotation. We achieve this by replacing regular CNN filters with circular harmonics, returning a maximal response and orientation for every receptive field patch. H-Nets use a rich, parameter-efficient and fixed computational complexity representation, and we show that deep feature maps within the network encode complicated rotational invariants. We demonstrate that our layers are general enough to be used in conjunction with the latest architectures and techniques, such as deep supervision and batch normalization. We also achieve state-of-the-art classification on rotated-MNIST, and competitive results on other benchmark challenges.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Making Convolutional Networks Shift-Invariant Again (2019, background): “Recent work has strived to build in additional types of in-variances, such as rotation, reﬂection, and scaling (Sifre & Mallat, 2013; Bruna & Mallat, 2013; Esteves et al., 2018; Kanazawa et al., 2014; Worrall et al., 2017; Cohen & Welling, 2016).”
Rotation Equivariant CNNs for Digital Pathology (2018, methodology): “A standard convolution 2 (denoted ∗ ) of an input f with ﬁlter ψ is deﬁned as: G-CNNs are a generalization of CNNs that are equivariant under more general symmetry groups, such as the group G = p 4 of 90 ◦ rotations, or G = p 4 m which additionally includes reﬂection.”
Deformable Convolutional Networks (2017, background): “Some works are devoted for speciﬁc transformations such as symmetry [13, 9], scale [29], and rotation [53].”
Why do deep convolutional networks generalize so poorly to small image transformations? (2018, methodology): “Alternatively, one could use specially designed features such as SIFT in which invariance is hard coded [7, 29, 30, 31] or neural network architectures that explicitly enforce invariance [36, 19, 5, 6, 11, 12, 47, 46, 8].”
Covariant Compositional Networks For Learning Graphs (2018, background): “This is the simplest manifestation of a well studied property of CNNs called equivariance (Cohen & Welling, 2016a; Worrall et al., 2017).”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “…equivariance of CNNs has been extended to additional symmetries ranging from planar rotations (Dieleman et al., 2016; Marcos et al., 2017; Worrall et al., 2017; Weiler et al., 2018b; Li et al., 2018; Cheng et al., 2018; Hoogeboom et al., 2018; Bekkers et al., 2018; Veeling et al., 2018;…”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “Another approach, drawing on harmonic analysis, ﬁnds a basis of equivariant functions and parametrizes convolutional kernels in that basis (Worrall et al., 2017; Weiler and Cesa, 2019).”
Set2Graph: Learning Graphs From Sets (2020, background): “…et al., 2018; Maron et al., 2019b;a), learning images with rotational and reﬂectional symmetries (Cohen & Welling, 2016a;b; Dieleman et al., 2016; Worrall et al., 2017), learning functions on spheres (Cohen et al., 2018; Esteves et al., 2017) and learning general 3D data (Weiler et al., 2018;…”
General E(2)-Equivariant Steerable CNNs (2019, background): “…able to implement a wide range of equivariant models, covering regular GCNNs [6, 7, 8, 9, 10, 11], classical Steerable CNNs [1], Harmonic Networks [12], gated Harmonic Networks [2], Vector Field Networks [13], Scattering Transforms [14, 15, 16, 17, 18] and entirely new architectures, in one…”
Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds (2018, methodology): “This choice of filter restriction is analogous to the use of circular harmonics in Worrall et al. (2017) (though we do not have an analog to the phase offset because of the non-commutativity of SO(3)).”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, background): “, 2019), translations and rotations of the plane and 3D space (Oyallon & Mallat, 2015; Cohen & Welling, 2016; 2017; Marcos et al., 2017; Weiler et al., 2018b;a; Worrall et al., 2017; Worrall & Brostow, 2018; Winkels & Cohen, 2018; Winkens et al., 2018; Thomas et al., 2018; Bekkers et al., 2018), and”
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, background): “Spherical CNNs are not the only context in which the idea of Fourier space neural networks has recently appeared [18, 19, 5, 7].”
RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks (2018, methodology): “In particular, steer-able ﬁlters were used in [5, 36, 37].”
Spherical CNNs (2018, background): “With the exception of SO(2)-steerable networks (Worrall et al., 2017; Weiler et al., 2017), these networks are all limited to discrete groups, such as discrete rotations acting on planar images or permutations acting on point clouds.”
On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups (2018, methodology): “In particular, we argue for Fourier space representations, similar to those that have appeared in (Worrall et al., 2017; Cohen et al., 2018; Kondor et al., 2018)), and, even more recently, since the submission of the original version of the present paper in (Thomas et al.”
Alias-Free Generative Adversarial Networks (2021, background): “Group-equivariant CNNs aim to generalize the efﬁciency beneﬁts of translational weight sharing to, e.g., rotation [16, 65, 63, 62] and scale [64].”
Learning SO(3) Equivariant Representations with Spherical CNNs (2017, background): “Only recently, equivariance has been addressed with respect to other groups [1,2] and CNNs have been proposed for manifolds or graphs [3,4,5].”
Learning Steerable Filters for Rotation Equivariant CNNs (2017, background): “Worrall et al. [23] achieve continuous resolution in orientations by working with complex valued steerable ﬁlters and feature maps.”
Rotation Equivariant Vector Field Networks (2016, methodology): “01%, thus improving significantly over TIPooling and over the more recent H-Net [29] and ORN [31].”