Unsupervised Discovery of Parts, Structure, and Dynamics

By Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin P. Murphy, William T. Freeman, Josh Tenenbaum, Jiajun Wu. 2019

62 citations recorded by the source corpus.

Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future. In this paper, we propose a novel formulation that simultaneously learns a hierarchical, disentangled object representation and a dynamics model for object parts from unlabeled videos. Our Parts, Structure, and Dynamics (PSD) model learns to, first, recognize the object parts via a layered image representation; second, predict hierarchy via a structural descriptor that composes low-level concepts into a hierarchical structure; and third, model the system dynamics by predicting the future. Experiments on multiple real and synthetic datasets demonstrate that our PSD model works well on all three tasks: segmenting object parts, building their hierarchical structure, and capturing their motion distributions.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards causal generative scene models via competition of experts (2020, neutral): “First results in the direction of video data have been shown by Xu et al. (2019).”
Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery (2021, background): “Existing work extracts parts from pairs of images [53, 51], point clouds [55] or videos [41, 27, 25].”
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning (2021, background): “It benefits many practical applications and downstream tasks, such as the precipitation forecasting [1], [2], traffic flow prediction [3], [4], physical scene understanding [5], [6], [7], [8], early activity recognition [9], deep reinforcement learning [10], [11], and the vision-based model predicti”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, background): “…a disentangled latent state with a single function [3, 19–22] or processing each disentangled factor in a permutation-sensitive manner [3, 11, 23] (1) assumes a ﬁxed number of entities that cannot be dynamically adjusted for generalizing to more objects than in training and (2) has no…”
Unsupervised Learning of Object Structure and Dynamics from Videos (2019, background): “[29] proposed to discover object parts and structure by watching how they move in videos.”
Contrastive Learning of Structured World Models (2019, background): “Several recent works succeed in learning such structured models directly from pixels (Watters et al., 2017; van Steenkiste et al., 2018; Xu et al., 2019; Watters et al., 2019), but in contrast to our work rely on pixel-based loss functions.”
Learning Physical Graph Representations from Visual Scenes (2020, background): “While most related works on inferring scene graph representations from visual data require ﬁne-grained annotations, several works have studied inferring objects and their relations from videos without supervision [50, 43, 53].”
Unsupervised part representation by Flow Capsules (2020, methodology): “In particular, we evaluate the approach on image data of scenes containing multiple occluding colored geometrical shapes (Geo), Geo with ImageNet (Deng et al., 2009) background and textured objects (Geo + ), and on real human motions (Exercise) (Xu et al., 2019) } .”