Learning the Irreducible Representations of Commutative Lie Groups

By Taco Cohen, Max Welling. 2014

124 citations recorded by the source corpus.

We present a new probabilistic model of compact commutative Lie groups that produces invariant-equivariant and disentangled representations of data. To define the notion of disentangling, we borrow a fundamental principle from physics that is used to derive the elementary particles of a system from its symmetries. Our model employs a newfound Bayesian conjugacy relation that enables fully tractable probabilistic inference over compact commutative Lie groups - a class that includes the groups that describe the rotation and cyclic translation of images. We train the model on pairs of transformed image patches, and show that the learned invariant representation is highly effective for classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “The lack of almost any identifiability result in non-linear ICA has been a main bottleneck for the utility of the approach (Hyvarinen et al., 2019) and partially motivated alternative machine learning approaches (Desjardins et al., 2012; Schmidhuber, 1992; Cohen and Welling, 2014b).”
NICE: Non-linear Independent Components Estimation (2014, methodology): “Note that the architecture of our model could be trained using other inductive principles capable of exploiting its advantages, like toroidal subspace analysis (TSA) (Cohen and Welling, 2014).”
Deep Convolutional Inverse Graphics Network (2015, methodology): “[6] have considered this problem by proposing a theoretical framework to learn irreducible representations having both invariances and equiv-ariances, coming up with the best representation for any given task is an open question.”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Due to the initial poor performance of purely unsupervised methods, the field initially focused on semi-supervised [62, 11, 57, 58, 44, 46] and weakly supervised approaches [31, 12, 40, 21, 78, 20, 15, 35, 80, 54, 47, 64, 8].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “(Hinton et al., 2011; Cohen & Welling, 2014) assume some knowledge of the effect of the factors of variations even though they are not observed.”
Disentangling by Factorising (2018, methodology): “Given a task, it will be easy to obtain an invariant representation from the disentangled representation by ignoring the dimensions encoding the nuisance factors for the task (Cohen & Welling, 2014).”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “[15] proposed a deﬁnition based on group theory [10] which is (informally) stated as follows: “A representation z is disentangled w.r.t a particular subgroup y k (from a symmetry group y = { y k } =1 ) if z can be decomposed into different subspaces { z i } =1 in which the subspace z i should be…”
Towards a Definition of Disentangled Representations (2018, background): “Earlier work in this area mostly focused on learning transformations in observation space, such as translation and in-plane rotation of objects [62, 70, 19].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “) (Hinton et al., 2011; Cohen and Welling, 2014), (iii) knowledge about the changes in the generative factors across observations (e.”
Understanding disentangling in β-VAE (2018, background): “A number of purely unsupervised approaches to disentangled factor learning have been proposed [36, 10, 39, 8, 9, 6, 15], including β -VAE [15], the focus of this text. β -VAE is a state of the art model for unsupervised visual disentangled representation learning.”
Disentangling Factors of Variation Using Few Labels (2019, background): “Exploiting relational information or knowledge of the effect of the factors of variation have both been qualitatively studied to learn disentangled representations [23, 9, 30, 18, 63, 16, 13, 25, 65, 43, 35, 55, 4].”
Early Visual Concept Learning with Unsupervised Deep Learning (2016, background): “So far any purely unsupervised approaches to disentangled factor learning have not scaled well [11, 30, 9, 10].”
beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework (2016, background): “Reprinted with permission. unsupervised approaches to disentangled factor learning have not scaled well (Schmidhuber, 1992; Desjardins et al., 2012; Tang et al., 2013; Cohen & Welling, 2014; 2015).”
A Framework for the Quantitative Evaluation of Disentangled Representations (2018, background): “Properties such as invariance and equivariance have been proposed as desiderata for representations or codes (Goodfellow et al., 2009; Kivinen & Williams, 2011; Cohen & Welling, 2014b; Lenc & Vedaldi, 2015; Jayaraman & Grauman, 2015).”