Unsupervised Learning of Disentangled Representations from Video

By Emily L. Denton, Vighnesh Birodkar. 2017

583 citations recorded by the source corpus.

We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017; Yingzhen and Mandt, 2018b; Locatello et al., 2018).”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Subsequent works have used RNN-based models to generate many frames into the future [8, 11, 25, 60], and also as an internal model to reason about the future [68, 90, 102].”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “…a number of works rely on known relations between the factors of variation (Karaletsos et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Locatello et al., 2018) and disentangling motion and pose from content (Hsieh et…”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Due to the initial poor performance of purely unsupervised methods, the field initially focused on semi-supervised [62, 11, 57, 58, 44, 46] and weakly supervised approaches [31, 12, 40, 21, 78, 20, 15, 35, 80, 54, 47, 64, 8].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “One can also exploit known relations between factors in different samples [18, 52, 16, 13, 21, 54].”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, background): “ting to the number of objects currently present in an image. Another interesting line of work is the GAN-based unsupervised video generation that decomposes motion and content (Tulyakov et al., 2018; Denton and Birodkar, 2017). These methods learn interpretable features of content and motion, but de”
Disentangling by Factorising (2018, background): “…or explicit knowledge about the true underlying factors of the data have excelled at disentangling (Kulkarni et al., 2015; Kingma et al., 2014; Reed et al., 2014; Siddharth et al., 2017; Hinton et al., 2011; Mathieu et al., 2016; Goroshin et al., 2015; Hsu et al., 2017; Denton & Birodkar, 2017).”
Conditional variance penalties and domain shift robustness (2017, background): “Denton and Birodkar (2017) propose an autoencoder framework to disentangle style and content in videos using an adversarial loss term where the grouping structure induced by clip identity is exploited.”
Towards a Definition of Disentangled Representations (2018, background): “However, many other approaches [50, 23, 66, 63, 80, 78, 36, 17, 77, 46] and a recent disentanglement metric by [65] consider it perfectly ﬁne for each data generative factor to be encoded by multiple latent dimensions.”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, background): “Prediction has been an active area of research in computer vision, including prediction of future images [9, 10, 11], motion [12], and even physical events [13, 14, 15].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “Much of the earlier work assumes some sort of supervision in terms of: (i) partial or full access to the generative factors per instance (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Karaletsos et al., 2015), (ii) knowledge about the nature of generative factors (e.g, translation, ro”
Stochastic Variational Video Prediction (2017, background): “…the complex dynamics of real-world objects and people, and therefore is more likely to acquire a representation that can be used for a variety of visual perception tasks, such as object tracking and action recognition (Srivastava et al., 2015; Lotter et al., 2017; Denton & Birodkar, 2017).”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Somewhat related, Villegas et al. [70], Denton and Birodkar [71], Fraccaro et al. [72] propose autoencoder models for video sequence prediction with separate encoders disentangling the latent code into pose and content.”
Stochastic Video Generation with a Learned Prior (2018, background): “Denton & Birodkar (2017) decomposes frames into content and pose representations using a form of adversarial loss to give a clean separation.”
MoCoGAN: Decomposing Motion and Content for Video Generation (2017, background): “The future frame prediction problem studied in [34, 26, 24, 17, 10, 38, 44, 40, 7] is different to the video generation problem.”