Learning Physical Graph Representations from Visual Scenes

By Daniel M. Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Josh Tenenbaum and 1 more. 2020

88 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) have proved exceptional at learning representations for visual object categorization. However, CNNs do not explicitly encode objects, parts, and their physical properties, which has limited CNNs' success on tasks that require structured understanding of visual scenes. To overcome these limitations, we introduce the idea of Physical Scene Graphs (PSGs), which represent scenes as hierarchical graphs, with nodes in the hierarchy corresponding intuitively to object parts at different scales, and edges to physical connections between parts. Bound to each node is a vector of latent attributes that intuitively represent object properties such as surface shape and texture. We also describe PSGNet, a network architecture that learns to extract PSGs by reconstructing scenes through a PSG-structured bottleneck. PSGNet augments standard CNNs by including: recurrent feedback connections to combine low and high-level image information; graph pooling and vectorization operations that convert spatially-uniform feature maps into object-centric graph structures; and perceptual grouping principles to encourage the identification of meaningful scene elements. We show that PSGNet outperforms alternative self-supervised scene representation algorithms at scene segmentation tasks, especially on complex real-world images, and generalizes well to unseen object types and scene arrangements. PSGNet is also able learn from physical motion, enhancing scene estimates even for static images. We present a series of ablation studies illustrating the importance of each component of the PSGNet architecture, analyses showing that learned latent attributes capture intuitive scene properties, and illustrate the use of PSGs for compositional scene inference.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Notice that, since clustering-based approaches to image segmentation focus on low-level similarity structures, their understanding of objects at a more high-level is limited (i.e. at the level of object representations, but see Bear et al., 2020).”
How to Represent Part-Whole Hierarchies in a Neural Network (2021, background): “Combining this type of dynamically created graph with neural network learning techniques has recently shown great promise [Bear et al., 2020], but if the whole computer is a neural network, it is far less obvious how to represent part-whole hierarchies that are diﬀerent for every image if we want…”
BlobGAN: Spatially Disentangled Scene Representations (2022, background): “semantic segmentation maps [57, 44], object-attribute graphs [32, 8], text [99, 66, 52, 64, 69, 65], pose [42, 73, 3], and keypoints [11].”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “In terms of what we consider to present the most opportunity for future work, we note that there are relatively few continuous optimization approaches which seek to learn structured, semantic representations from nonsemantic, high-dimensional data such as video or image data (exceptions include Caus”
Unsupervised part representation by Flow Capsules (2020, background): “Most recently, (Bear et al., 2020; Veerapaneni et al., 2020) extend such networks to incorporate motion, but still rely on iterative refinement.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Based on such benchmarks, some works [70, 11, 48, 52] learn compositional scene structure and estimate states through physical motions and visual de-animation.”
Conditional Object-Centric Learning from Video (2021, background): “PSGNet (Bear et al., 2020) learns to decompose static images or individual frames from a video into hierarchical scene graphs using motion information estimated from neighboring video frames.”