Learning to Decompose and Disentangle Representations for Video Prediction

By Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, Juan Carlos Niebles. 2018

322 citations recorded by the source corpus.

Our goal is to predict future video frames given a sequence of input frames. Despite large amounts of video data, this remains a challenging task because of the high-dimensionality of video frames. We address this challenge by proposing the Decompositional Disentangled Predictive Auto-Encoder (DDPAE), a framework that combines structured probabilistic models and deep networks to automatically (i) decompose the high-dimensional video that we aim to predict into components, and (ii) disentangle each component to have low-dimensional temporal dynamics that are easier to predict. Crucially, with an appropriately specified generative model of video frames, our DDPAE is able to learn both the latent decomposition and disentanglement without explicit supervision. For the Moving MNIST dataset, we show that DDPAE is able to recover the underlying components (individual digits) and disentanglement (appearance and location) as we would intuitively do. We further demonstrate that DDPAE can be applied to the Bouncing Balls dataset involving complex interactions between multiple objects to predict the video frame directly from the pixels and recover physical states without explicit supervision.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “There is for example a rich literature in disentangling pose from content in 3D objects and content from motion in videos or time series in general (Yang et al., 2015; Yingzhen and Mandt, 2018a; Hsieh et al., 2018; Fortuin et al., 2019; Deng et al., 2017; Goroshin et al., 2015).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “…(Karaletsos et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Locatello et al., 2018) and disentangling motion and pose from content (Hsieh et al., 2018; Fortuin et al., 2019; Deng et al., 2017; Goroshin et al., 2015).”
On the Fairness of Disentangled Representations (2019, background): “Disentangling pose from content and content from motion are also classical computer vision problems that have been tackled with various degrees of supervision and inductive bias [92, 93, 40, 25, 18, 31, 42].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Disentangling pose, style, or motion from content are classical vision tasks that has been studied with different degrees of supervision [72, 79, 80, 34, 19, 14, 21, 36].”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Hsieh et al. [69] use a similar partition of the latent code, but additionally allow the model to decompose the input into different parts, e.g., modelling different moving objects in a video sequence.”
Disentangling Factors of Variation Using Few Labels (2019, background): “There is for example a rich literature in disentangling pose from content in 3D objects and content from motion in videos [64, 65, 24, 15, 12, 18, 26].”
Scalable Object-Oriented Sequential Generative Models (2019, background): “DDPAE (Hsieh et al., 2018) is another seqeuntial generative model for object-oriented learning, which models each object in the scene with an appearance and a position vector.”
Deep Learning for Anomaly Detection (2020, methodology): “This technique is widely used for sequence representation learning and prediction [63, 82, 98, 146].”
Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking (2019, unknown): “Similar extensions are also provided by (Hsieh et al. 2018; He et al. 2018).”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “Other disentangling approaches factorize the video into independent components [60, 58, 12, 21, 19].”
CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models (2020, background): “Disentangled representation learning is of great importance in various applications such as computer vision, speech and natural language processing, and recommender systems [9, 20, 8].”
Disentangling User Interest and Conformity for Recommendation with Causal Embedding (2021, background): “Existing approaches mainly focus on computer vision [15, 16, 22, 23, 28].”
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning (2021, background): “Typical approaches include using optical flows [57], [58], adversarial training schemes [59], reasoning about object-centric content and pose vectors [60], [61], differ-”
SimVP: Simpler yet Better Video Prediction (2022, unknown): “2014-2015 [41, 55, 71] [36, 46] [40] 2016-2017 [38, 47, 63, 67] [3, 9, 15, 32, 62] [20, 35, 59] 2018-2019 [22, 43, 56, 65, 69, 74] [6, 10, 61, 66, 73] [70] [16, 29, 72] 2020-2021 [68] [18, 23] [48] [7, 54]”
Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views (2021, background): “While some works in multi-object discovery and tracking in videos appear to be MOMV models [16, 12], they in fact work with one view per scene (abiding strictly by our deﬁnition of a scene in Section 2) and are only capable of dealing with binarizable MNIST-like images.”
Latent Video Transformer (2020, unknown): “…and unconditional video synthesis have been addressed for a long time and the solutions include recurrent models [15, 47, 48, 4], VAE-based models [8, 11, 25, 20, 9], autoregressive models [33, 39, 50, 22, 49, 19], normalizing ﬂows [24], GANs [27, 44, 35, 40, 1, 36, 7, 26] and optical ﬂow [32, 30].”
Deep Stable Learning for Out-Of-Distribution Generalization (2021, background): “There are many works attempt learning disentangled features for robust or explainable representations[26, 6, 24] with strong constrains on features.”
R-SQAIR: Relational Sequential Attend, Infer, Repeat (2019, background): “Many recent models have AIR as the core mechanism: SQAIR [18] extends AIR to sequential settings, similarly does DDPAE [15].”