Equivariance Through Parameter-Sharing

By Siamak Ravanbakhsh, Jeff Schneider, Barnab'as P'oczos. 2017

290 citations recorded by the source corpus.

We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group G that acts discretely on the input and output of a standard neural network layer ϕW : ℝM → ℝN, we show that ϕW is equivariant with respect to G-action iff G explains the symmetries of the network parameters W. Inspired by this observation, we then propose two parameter-sharing schemes to induce the desirable symmetry on W. Our procedure for tying the parameters achieves G-equivariance and, under some conditions on the action of G, it guarantees sensitivity to all other permutation groups outside G.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Directional Message Passing for Molecular Graphs (2020, background): “In principle, these invariances can be learned by any neural network via corresponding weight matrix symmetries (Ravanbakhsh et al., 2017).”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “Consequently, Ravanbakhsh et al. (2017) introduced the concept of unique (cid:71) -equivariance to express the family of functions equivariant to (cid:71) but not equivariant to other groups (cid:71) ′ ≥ (cid:71) : In the following sections, we shown that we can enforce unique (cid:71)…”
Set2Graph: Learning Graphs From Sets (2020, methodology): “Except from designing invariant and equivariant networks there has been a keen interest in the analysis of such models (Ravanbakhsh et al., 2017; Kondor et al., 2018), especially the analysis of their approximation power (Zaheer et al., 2017; Qi et al., 2017; Maron et al., 2019c; Keriven & Peyr´e,…”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, neutral): “…Maron et al., 2019), translations and rotations of the plane and 3D space (Oyallon & Mallat, 2015; Cohen & Welling, 2016; 2017; Marcos et al., 2017; Weiler et al., 2018b;a; Worrall et al., 2017; Worrall & Brostow, 2018; Winkels & Cohen, 2018; Winkens et al., 2018; Thomas et al., 2018; Bekkers…”
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, background): “Recently, a number of papers have appeared that examine equivariance from the theoretical point of view, motivated by the understanding that the natural way to generalize convolutional networks to other types of data will likely lead through generalizing the notion of equivariance itself to other tr”
On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups (2018, background): “One work that is close in spirit to the present paper but only considers discrete groups is (Ravanbakhsh et al., 2017).”
A General Theory of Equivariant CNNs on Homogeneous Spaces (2018, methodology): “A framework for (non-convolutional) networks equivariant to finite groups was presented by [17], and equivariant set and graph networks are analyzed by [18–21].”
Inductive biases for deep learning of higher-level cognition (2020, background): “, 2020), invariance or equivariance to known transformations (Bruna et al., 2013; Defferrard et al., 2016; Ravanbakhsh et al., 2017; Thomas et al., 2018; Finzi et al., 2020; Satorras et al., 2021) or choices of prior distributions in a Bayesian model (Jeffreys, 1946; Berger and Bernardo,”
A Group-Theoretic Framework for Data Augmentation (2019, background): “Starting from [30, 45, 61, 14], a line of research focuses on designing (or learning) neural net architectures that are invariant to certain symmetries [31, 27, 70, 20, 22, 19], equivariant to SO(3) group [21, 28, 29], and others [58, 42, 69, 41].”
Translational Equivariance in Kernelizable Attention (2021, methodology): “…that x i = e i + u i , the pre-attention matrix computation can be described with Strategy 1 In our ﬁrst strategy, we discard interaction terms between the content embeddings e and the absolute positions u in Equation 2 and follow the notion, that equivariance is linked to parameter sharing [15].”
Universal Equivariant Multilayer Perceptrons (2020, methodology): “[31] Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos.”
Probabilistic Symmetries and Invariant Neural Networks (2020, background): “…of feed-forward networks (i.e., linear maps composed with pointwise nonlinearities) that are invariant under the action of discrete groups (Shawe-Taylor, 1989; Ravanbakhsh et al., 2017), finite linear groups (Wood and Shawe-Taylor, 1996), or compact groups (Kondor and Trivedi, 2018).”
Graph Neural Networks for Scalable Radio Resource Management: Architecture Design and Theoretical Analysis (2020, background): “training, and better generalization empirically [14], [21], [22], [28] and provably [23].”
Scale equivariance in CNNs with vector fields (2018, background): “Equivariance to other transformations, chieﬂy rotation, through weight sharing has gathered a lot of attention in recent literature (Cohen & Welling, 2016; Dieleman et al., 2016; Ravanbakhsh et al., 2017; Weiler et al., 2017; Zhou et al., 2017).”
Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs (2018, background): “Ravanbakhsh et al. (2017b) provides a weight-sharing scheme for maintaining general neural network equivariances characterized as automorphisms of a colored multi-edged bipartite graph.”
On the Universality of Invariant Networks (2019, methodology): “These equations are equivalently formulated using weight sharing scheme in (Ravanbakhsh et al., 2017).”
Permutation Equivariant Models for Compositional Generalization in Language (2020, methodology): “Rather, equivariance for this module is achieved via parameter-sharing (Ravanbakhsh et al., 2017).”