3D-Aware Scene Manipulation via Inverse Graphics

By Shunyu Yao, Tzu-Ming Harry Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, Josh Tenenbaum. 2018

92 citations recorded by the source corpus.

We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

MONet: Unsupervised Scene Decomposition and Representation (2019, background): “Methods in this camp [Tian et al., 2019, Yao et al., 2018] make domain-speciﬁc assumptions about the latent code [Kulkarni et al., 2015] or generative process [Wu et al., 2017] to keep inference tractable, but these assumptions tend to be too strong for the methods to be broadly useful.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “Considering the cubic complexity of the voxel representation, recent papers have explored meshes [Groueix et al., 2018, Yao et al., 2018] and point clouds [Fan et al., 2017] in the context of neural networks.”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, background): “For example, Yao et al. [11] segment objects and subsequently render them in different poses.”
Learning to Manipulate Individual Objects in an Image (2020, methodology): “In [37], an image is decomposed into semantic mask, texture, and geometry, requiring heavy manual supervision.”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, background): “This can be achieved with explicit 3D representations, like appearance ﬂow [58], occupancy voxel grids [43, 59], meshes, or shape templates [27, 46, 56], in conjunction with handcrafted differentiable renderers [8, 17, 31, 33].”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, methodology): “(15) Similar to [12, 34, 40], we additionally use a perceptual loss and discriminator feature matching loss to make training more stable:”
Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views (2021, unknown): “To avoid the additional computational complexities of factoris-ing or segmenting the objects in a scene into explicit multi-object representations, many works have used pre-segmented images [26, 27].”
BlobGAN: Spatially Disentangled Scene Representations (2022, methodology): “With the advancement of deep generative models, parametric analysisby-synthesis techniques are having a renaissance, with some top-down [93, 55, 95] as well as bottom-up [60, 22] techniques.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “multiple views of the same object [45], videos [29] or some pre-existing 3D mesh representation that is the starting point for further disentanglement [13, 35, 53, 40] or self-supervision [56].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “cGANs have been used for convolutional face generation [142], face aging [143], image translation [144], synthesizing outdoor images having specific scenery attributes [145], natural image description [146], and 3D-aware scene manipulation [147].”
Learning Physical Graph Representations from Visual Scenes (2020, methodology): “The most notable approaches include Factor3D [51], MeshRCNN [15], and 3D-SDN [57].”