Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views

By Nanbo Li, Cian Eastwood, Robert B. Fisher. 2021

62 citations recorded by the source corpus.

Learning object-centric representations of multi-object scenes is a promising approach towards machine intelligence, facilitating high-level reasoning and control from visual sensory data. However, current approaches for unsupervised object-centric scene representation are incapable of aggregating information from multiple observations of a scene. As a result, these"single-view"methods form their representations of a 3D scene based only on a single 2D observation (view). Naturally, this leads to several inaccuracies, with these methods falling victim to single-view spatial ambiguities. To address this, we propose The Multi-View and Multi-Object Network (MulMON) -- a method for learning accurate, object-centric representations of multi-object scenes by leveraging multiple views. In order to sidestep the main technical difficulty of the multi-object-multi-view scenario -- maintaining object correspondences across views -- MulMON iteratively updates the latent object representations for a scene over multiple views. To ensure that these iterative updates do indeed aggregate spatial information to form a complete 3D scene understanding, MulMON is asked to predict the appearance of the scene from novel viewpoints during training. Through experiments, we show that MulMON better-resolves spatial ambiguities than single-view methods -- learning more accurate and disentangled object representations -- and also achieves new functionality in predicting object segmentations for novel viewpoints.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Further improvements can be made by assuming access to multiple different views to explicitly model 3D structure at a representational level (Chen et al., 2020; Nanbo et al., 2020).”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, methodology): “Examples of hybrid approaches include methods such as Recurrent Independent Mechanisms [76], graph networks [11, 12, 213], and a large body of work on scene understanding [116, 155, 171, 173, 270].”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Recent works therefore investigate how the synthesis process can be controlled at the object-level [3,4,7,18,19,26,45,86,90].”