Transformation Properties of Learned Visual Representations

By Taco Cohen, Max Welling. 2014

109 citations recorded by the source corpus.

When a three-dimensional object moves relative to an observer, a change occurs on the observer's image plane and in the visual representation computed by a learned model. Starting with the idea that a good visual representation is one that transforms linearly under scene motions, we show, using the theory of group representations, that any such representation is equivalent to a combination of the elementary irreducible representations. We derive a striking relationship between irreducibility and the statistical dependency structure of the representation, by showing that under restricted conditions, irreducible representations are decorrelated. Under partial observability, as induced by the perspective projection of a scene onto the image plane, the motion group does not have a linear action on the space of images, so that it becomes necessary to perform inference over a latent representation that does transform linearly. This idea is demonstrated in a model of rotating NORB objects that employs a latent representation of the non-commutative 3D rotation group SO(3).

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “The lack of almost any identifiability result in non-linear ICA has been a main bottleneck for the utility of the approach (Hyvarinen et al., 2019) and partially motivated alternative machine learning approaches (Desjardins et al., 2012; Schmidhuber, 1992; Cohen and Welling, 2014b).”
Spatial Transformer Networks (2015, methodology): “Cohen & Welling [6] analyse this behaviour in relation to symmetry groups, which is also exploited in the architecture proposed by Gens & Domingos [10], resulting in feature maps that are more invariant to symmetry groups.”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, background): “This includes the development of transforming auto-encoders [28] and recent probabilistic models for images[29, 30].”
Why do deep convolutional networks generalize so poorly to small image transformations? (2018, methodology): “also defined a weaker form of translation invariance, which they called "shiftability" and showed that it can hold for systems with subsampling (this is related to weak translation invariance as defined by [28], see also [14, 9] for related ideas applied to neural networks).”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “The lack of almost any identifiability result in non-linear ICA has been a main bottleneck for the utility of the approach [24] and partially motivated alternative machine learning approaches [14, 46, 11].”
Towards a Definition of Disentangled Representations (2018, background): “Later work extended this to general 3D rotations [20], but did not learn to disentangle this rotation from other factors of variation.”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “It is a challenging problem and many of the earlier attempts have not been able to scale well for realistic settings (Schmidhuber, 1992; Desjardins et al., 2012; Cohen and Welling, 2015) (see also, Higgins et al. (2017)).”
Understanding disentangling in β-VAE (2018, background): “tors [18, 35, 32, 44, 43, 12, 24, 7, 42, 20]. This, however, is unrealistic in most real world scenarios. A number of purely unsupervised approaches to disentangled factor learning have been proposed [36, 10, 39, 8, 9, 6, 15], including -VAE [15], the focus of this text. -VAE is a state of the art m”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…of the world is addressed by the large body of literature on disentangled representation learning (Schmidhuber, 1992; Desjardins et al., 2012; Cohen & Welling, 2014; 2015; Kulkarni et al., 2015; Hinton et al., 2011; Rippel & Adams, 2013; Reed et al., 2014; Yang et al., 2015; Goroshin et al.,…”
Learning to Linearize Under Uncertainty (2015, background): “Theoretical properties of linearizing features were studied in [3].”
On Invariance and Selectivity in Representation Learning (2015, neutral): “Some other relevant works are [10,11,17,28,36].”
Intertwiners between Induced Representations (with Applications to the Theory of Equivariant Neural Networks) (2018, methodology): “In representation learning, the idea of irreducible representations as elementary particles has been applied to formalize the idea of “disentangling” or “capsules” that represent distinct visual entities [Cohen and Welling, 2014, 2015], each of which has a certain type [Cohen and Welling, 2017].”
A Framework for the Quantitative Evaluation of Disentangled Representations (2018, background): “Properties such as invariance and equivariance have been proposed as desiderata for representations or codes (Goodfellow et al., 2009; Kivinen & Williams, 2011; Cohen & Welling, 2014b; Lenc & Vedaldi, 2015; Jayaraman & Grauman, 2015).”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, background): “Some prior work infers voxel grid representations of 3D scenes from images [2, 4, 5] or uses them for 3D-structure-aware generative models [6, 33].”
Deferred neural rendering (2019, unknown): “Images can be synthesized based on low-dimensional feature vectors [Eslami et al. 2018; Yan et al. 2016] that obey geometric constraints [Cohen and Welling 2014; Worrall et al. 2017].”
beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework (2016, background): “Reprinted with permission. unsupervised approaches to disentangled factor learning have not scaled well (Schmidhuber, 1992; Desjardins et al., 2012; Tang et al., 2013; Cohen & Welling, 2014; 2015).”
What Happened to My Dog in That Network: Unraveling Top-down Generators in Convolutional Neural Networks (2015, background): “Deep computer vision Two recent related works in computer vision are (Cohen & Welling, 2015; Jaderberg et al., 2015).”