Learning Stable Group Invariant Representations with Convolutional Networks

By Joan Bruna, Arthur Szlam, Yann LeCun. 2013

39 citations recorded by the source corpus.

Transformation groups, such as translations or rotations, effectively express part of the variability observed in many recognition problems. The group structure enables the construction of invariant signal representations with appealing mathematical properties, where convolutions, together with pooling operators, bring stability to additive and geometric perturbations of the input. Whereas physical transformation groups are ubiquitous in image and audio applications, they do not account for all the variability of complex signal classes. We show that the invariance properties built by deep convolutional networks can be cast as a form of stable group invariance. The network wiring architecture determines the invariance group, while the trainable filter coefficients characterize the group action. We give explanatory examples which illustrate how the network architecture controls the resulting invariance group. We also explore the principle by which additional convolutional layers induce a group factorization enabling more abstract, powerful invariant representations.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Scattering Networks for Hybrid Representation Learning (2018, background): “Recent works [12], [27], [36] have suggested that deep networks could build an approximation of the group of symmetries of a classification task and apply transformations along the orbits of this group, like convolutions.”
Robust Large Margin Deep Neural Networks (2016, neutral): “2708039 transformations such as translation [6], [7].”
Understanding deep convolutional networks (2016, methodology): “It is used by joint scattering transforms [29, 3] and has been proposed for unsupervised convolution network learning [9].”
Transformation Properties of Learned Visual Representations (2014, methodology): “ial observability is to model all variability that is not caused by the linear action of a low-dimensional Lie group as being caused by the action of the inﬁnite-dimensional group of diffeomorphisms (Bruna et al., 2013; Soatto, 2012). The scattering representations of Bruna & Mallat (2013) achie”
Learning SO(3) Equivariant Representations with Spherical CNNs (2017, methodology): “There are two threads within this space, spectral (Bruna et al. 2013b; Defferrard et al. 2016; Kipf and Welling 2016; Yi et al. 2016) and spatial (Boscaini et al. 2016; Masci et al. 2015; Monti et al. 2016; Wang et al. 2018).”
Scaling the Scattering Transform: Deep Hybrid Networks (2017, neutral): “Recent works [25, 7, 17] have suggested that deep networks could build an approximation of the group of symmetries of a classification task and apply transformations along the orbits of this group, like convolutions.”
Building a Regular Decision Boundary with Deep Networks (2017, background): “Geometric variabilities can be successfully handled by prior representations such as scattering transforms [23, 4, 20, 19, 25].”
Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy? (2015, background): “showed that the pooling stage in DNN result in shift-invariance [21].”
Generalization Error of Invariant Classifiers (2016, methodology): “Note that the theory from Section 3 does not apply directly as (i) the CNN trained without data augmentation is already (partially) invariant to translations due to its convolutional structure with pooling (Boureau et al., 2010; Bruna et al., 2013); (ii) Table 1: Classification accuracy [%] on CIFAR”
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations (2017, unknown): “…to groups of transformations was also studied for more classical convolutional neural networks from methodological and empirical points of view (Bruna et al., 2013; Cohen and Welling, 2016), and for shallow learned representations (Anselmi et al., 2016) or kernel methods (Haasdonk and…”
Invariance and Stability of Deep Convolutional Representations (2017, unknown): “Invariance to groups of transformations was also studied for more classical convolutional neural networks from methodological and empirical points of view [6, 9], and for shallow learned representations [1] or kernel methods [13, 19, 22].”