Unsupervised Learning of Object Structure and Dynamics from Videos

By Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P. Murphy, Honglak Lee. 2019

162 citations recorded by the source corpus.

Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning. To address this challenge, we adopt a keypoint-based image representation and learn a stochastic dynamics model of the keypoints. Future frames are reconstructed from the keypoints and a reference frame. By modeling dynamics in the keypoint coordinate space, we achieve stable learning and avoid compounding of errors in pixel space. Our method improves upon unstructured representations both for pixel-level video prediction and for downstream tasks requiring object-level understanding of motion dynamics. We evaluate our model on diverse datasets: a multi-agent sports dataset, the Human3.6M dataset, and datasets based on continuous control tasks from the DeepMind Control Suite. The spatially structured representation outperforms unstructured representations on a range of motion-related tasks such as object tracking, action recognition and reward prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Recently, there has been renewed interest in saliency-based approaches through the discovery of keypoints (Jakab et al., 2018; Kulkarni et al., 2019; Minderer et al., 2019; Gopalakrishnan et al., 2020).”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “In the video prediction context, VAEs are the foundation of many probabilistic models dealing with future uncertainty [9], [38], [55], [81], [85], [96], [97].”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, background): “Dynamics models describe sequential progress using a combination of stochastic and deterministic states in a latent space, which promotes an effective tracking and forecasting (Minderer et al., 2019; Sun et al., 2019; Wang et al., 2019a).”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, methodology): “predictive model, based on the “Structural VRNN” architecture proposed in [31], which is also closely related to approaches proposed in [32], [33].”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “[54] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P.”
The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation (2020, background): “his is exemplied by our task of pushing piles of small objects, or manipulating deformable objects such as clothes or uids. Similarly, other works have focused on identifying dynamics over keypoints [19,20], but it is not clear how to generalize keypoints to this problem where there are multiple sma”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, unknown): “2019a), and video prediction (Liu et al. 2018; Minderer et al. 2019).”
Visual Grounding of Learned Physical Models (2020, unknown): “Other researchers have shown better performance with intermediate representations like instance masks (Fragkiadaki et al., 2016; Watters et al., 2017; Janner et al., 2019), ob-Visual ject keypoints (Minderer et al., 2019), or dense visual descriptors (Xu et al., 2019).”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, methodology): “Of particular interest are methods based on learned object keypoints that correspond to highly informative (salient) regions in the image as indicated by the presence of object parts (Zhang et al., 2018; Jakab et al., 2018; Kulkarni et al., 2019; Minderer et al., 2019).”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “with human pose [61, 63] or key points [41], which imposes a severe overhead on the annotation budget.”
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis (2021, methodology): “However, also more complex models based on transformers [75], pixel-level autoregression [51, 40, 45, 16, 70, 10], factorization of dynamics and content [53, 24] and image warping using optical flow [71, 44, 25] have been proposed.”