Understanding Image Representations by Measuring Their Equivariance and Equivalence

By Karel Lenc, Andrea Vedaldi. 2014

632 citations recorded by the source corpus.

Despite the importance of image representations such as histograms of oriented gradients and deep Convolutional Neural Networks (CNN), our theoretical understanding of them remains limited. Aimed at filling this gap, we investigate two key mathematical properties of representations: equivariance and equivalence. Equivariance studies how transformations of the input image are encoded by the representation, invariance being a special case where a transformation has no effect. Equivalence studies whether two representations, for example two different parameterizations of a CNN, two different layers, or two different CNN architectures, share the same visual information or not. A number of methods to establish these properties empirically are proposed, including introducing transformation and stitching layers in CNNs. These methods are then applied to popular representations to reveal insightful aspects of their structure, including clarifying at which layers in a CNN certain geometric invariances are achieved and how various CNN architectures differ. We identify several predictors of geometric and architectural compatibility, including the spatial resolution of the representation and the complexity and depth of the models. While the focus of the paper is theoretical, direct applications to structured-output regression are demonstrated too.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “, 2016) while being independent from the task at hand (Goodfellow et al., 2009; Lenc and Vedaldi, 2015).”
Making Convolutional Networks Shift-Invariant Again (2019, background): “…(or robustness to changes) in response to manually generated perturbations to the input, such as image transformations (Goodfellow et al., 2009; Lenc & Vedaldi, 2015; Azulay & Weiss, 2019), geometric transforms (Fawzi & Frossard, 2015; Ruderman et al., 2018), and CG renderings with various…”
Rotation Equivariant CNNs for Digital Pathology (2018, background): “90-rotated and mirrored versions to improve robustness to orientation-induced instability. As a downside, this comes at 8 times the computational cost and does not provide guarantees on equivariance [11]. Methods that enable equivariance under rotations and other transformations include Harmonic Net”
Deformable Convolutional Networks (2017, background): “The invariance and equivalence of CNN representations to image transformations are studied in [36].”
Representation Learning by Learning to Count (2017, background): “The same procedure that we introduce is therefore applicable to a broader range of tasks as long as it is possible to express the transformation in feature space caused by a transformation in image space [24].”
Why do deep convolutional networks generalize so poorly to small image transformations? (2018, methodology): “also defined a weaker form of translation invariance, which they called "shiftability" and showed that it can hold for systems with subsampling (this is related to weak translation invariance as defined by [28], see also [14, 9] for related ideas applied to neural networks).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “Learning well-disentangled representations of complex sensory data has been identified as one of the key challenges in the quest for AI [1, 40, 26, 2, 42, 24, 47], since they should contain all the information present in the observations in a compact and interpretable structure [1, 21, 6] while bein”
On the Fairness of Disentangled Representations (2019, background): “Indeed, isolating each independent factor of variation into the independent components of a representation vector should make it both interpretable and simplify downstream prediction tasks [6, 7, 29, 35, 56, 58, 60, 74, 82, 87, 89, 90, 1, 28].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “They should contain all the information present in x in a compact and interpretable structure (Bengio et al., 2013; Kulkarni et al., 2015; Chen et al., 2016) while being independent from the task at hand (Goodfellow et al., 2009; Lenc & Vedaldi, 2015).”
Disentangling by Factorising (2018, background): “Invariance and equivariance are frequently considered to be desirable properties of representations in the literature (Goodfellow et al., 2009; Kivinen & Williams, 2011; Lenc & Vedaldi, 2015).”
Group Equivariant Convolutional Networks (2016, background): “Lenc & Vedaldi (2015) show that the AlexNet CNN (Krizhevsky et al., 2012) trained on imagenet spontaneously learns representations that are equivariant to ﬂips, scaling and rotation.”
Equivariance Through Parameter-Sharing (2017, background): “Deep and hierarchical models have used a variety of techniques to study or obtain representations that isolate transformations from the content “content” (Hinton et al., 2011; Jayaraman & Grauman, 2015; Lenc & Vedaldi, 2015; Agrawal et al., 2015, e.g. , ).”
Cross Modal Distillation for Supervision Transfer (2015, methodology): “Note that in obtaining these performance improvements we are using exactly the same CNN architecture and amount of labeled data.”
Harmonic Networks: Deep Translation and Rotation Equivariance (2016, background): “With data augmentation CNNs may learn some rotation equivariance, but this is difficult to quantify [21].”
Exploiting Cyclic Symmetry in Convolutional Neural Networks (2016, methodology): “The tasks which we wish to perform on such data with neural networks usually require equivari-ance to rotations: when the input is rotated, the learnt representations should change in a predictable way (Lenc & Vedaldi, 2014).”
Learning SO(3) Equivariant Representations with Spherical CNNs (2017, methodology): “[18] use a rotational ﬁlter orbit to produce oriented feature maps and rotationally invariant features, and Lenc and Vedaldi [19] propose a transformation layer which acts as a group-convolution by ﬁrst permuting then transforming by a linear ﬁlter.”
Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (2015, background): “Equivariance has been found to be useful for visual representations [15, 35, 25, 17].”
Analyzing Classifiers: Fisher Vectors and Deep Neural Networks (2015, background): “In recent years, interest in understanding image representations [15, 17, 19] and being able to explain the decision process of a classification system has increased, with e.”