A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning

By Marco Fraccaro, Simon Kamronn, Ulrich Paquet, Ole Winther. 2017

320 citations recorded by the source corpus.

This paper takes a step towards temporal reasoning in a dynamically changing video, not in the pixel space that constitutes its frames, but in a latent space that describes the non-linear dynamics of the objects in its world. We introduce the Kalman variational auto-encoder, a framework for unsupervised learning of sequential data that disentangles two latent representations: an object's representation, coming from a recognition model, and a latent state describing its dynamics. As a result, the evolution of the world can be imagined and missing data imputed, both without the need to generate high dimensional frames at each time step. The model is trained end-to-end on videos of a variety of simulated physical systems, and outperforms competing methods in generative and missing data imputation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017; Yingzhen and Mandt, 2018b; Locatello et al., 2018).”
A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “Similarly, probabilistic deep state space models retrieve data uncertainty in sequential data, and use it for learning-based ﬁltering [11], [12].”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “For example, a number of works rely on known relations between the factors of variation (Karaletsos et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Locatello et al., 2018) and disentangling motion and pose from content (Hs”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Due to the initial poor performance of purely unsupervised methods, the field initially focused on semi-supervised [62, 11, 57, 58, 44, 46] and weakly supervised approaches [31, 12, 40, 21, 78, 20, 15, 35, 80, 54, 47, 64, 8].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018) or explicit inductive biases (Locatello et al.”
Deep Factors for Forecasting (2019, background): “Motivated by (Fraccaro et al., 2017), we choose the following structural approximation q φ ( h , u | z ) := q φ ( u | z ) p ( h | u ) , where the second term matches the exact conditional posterior with the random effect probabilistic model R .”
Enhancing Time-Series Momentum Strategies Using Deep Neural Networks (2019, background): “the exponential smoothing RNN [18], autoregressive CNNs [19] and Kalman filter variants [20, 21] – while also making outputs easier to interpret by practitioners.”
Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, methodology): “The Kalman variational autoencoder (KVAE) (Fraccaro et al., 2017) extends ideas from the SVAE, modelling latent state using a linear Gaussian state space model (LGSSM).”
Meta-learning autoencoders for few-shot prediction (2018, background): “This class of problems is important in many areas, for example, learning and predicting physics [8, 26, 25, 3], reinforcement learning of games [15, 10], unsupervised learning of videos [20], and applications such as self-driving, where we cannot enumerate and train with all environments that the…”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “Much of the earlier work assumes some sort of supervision in terms of: (i) partial or full access to the generative factors per instance (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Karaletsos et al., 2015), (ii) knowledge about the nature of generative factors (e.g, translation, ro”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Somewhat related, Villegas et al. [70], Denton and Birodkar [71], Fraccaro et al. [72] propose autoencoder models for video sequence prediction with separate encoders disentangling the latent code into pose and content.”
Disentangled Sequential Autoencoder (2018, methodology): “We follow Fraccaro et al. (2017) to simulate video sequences of a ball (or a square) bouncing inside an irregular polygon us-ing Pymunk.”
Disentangling Factors of Variation Using Few Labels (2019, background): “[16] Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther.”
Generative Temporal Models with Spatial Memory for Partially Observed Environments (2018, methodology): “As shown in Figure 1a, in a Generative Temporal Model with Spatial Memory (GTM-SM) we introduce two sets of latent variables that disentangle visual and dynamics information, similarly to (Fraccaro et al., 2017).”
A General Method for Amortizing Variational Filtering (2018, background): “Amortized variational inference [27, 37] has enabled many recently proposed probabilistic deep dynamical latent variable models, with applications to video [42, 26, 43, 23, 15, 11, 3, 9, 29, 19], speech [7, 12, 17, 22, 29], handwriting [7], music [12], etc.”
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model (2019, background): “[17] M.”
Deep Factors with Gaussian Processes for Forecasting (2018, background): “The combination of probabilistic graphical models with deep neural networks has been an active research area recently [10, 12, 13, 14].”