Visual Object Networks: Image Generation with Disentangled 3D Representations

By Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, Bill Freeman. 2018

273 citations recorded by the source corpus.

Recent progress in deep generative models has led to tremendous breakthroughs in image generation. While being able to synthesize photorealistic images, existing models lack an understanding of our underlying 3D world. Different from previous works built on 2D datasets and models, we present a new generative model, Visual Object Networks (VONs), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel the image formation process into three conditionally independent factors---shape, viewpoint, and texture---and present an end-to-end adversarial learning framework that jointly models 3D shape and 2D texture. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e., silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic textures to these 2.5D sketches to generate realistic images. The VON not only generates images that are more realistic than the state-of-the-art 2D image synthesis methods but also enables many 3D operations such as changing the viewpoint of a generated image, shape and texture editing, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016, 2017, 2018, Zhu et al., 2018, Yan et al., 2016].”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “tive modeling by replicating the inner workings of graphics engines in deep networks. On the synthesis side, geometrydriven generative models using intrinsic images [59,5,57] or the 2.5D image sketch [87] as inputs to image synthesis networks have been shown to deliver sharper, more controllable ima”
Texture Fields: Learning Texture Representations in Function Space (2019, methodology): “Recent work has shown that image generation methods [3, 11, 19, 20, 24, 30, 47] can be improved by exploiting 3D knowledge about the generated shapes [1, 2, 49].”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “Some of these methods start from 3D representations [62, 1, 69, 44].”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, background): “Some prior work infers voxel grid representations of 3D scenes from images [6, 8, 9] or uses them for 3D-structure-aware generative models [10, 36].”
Editing Conditional Radiance Fields (2021, background): “Prior work reasons over an explicit [11, 15, 22, 72] or discrete volumetric [20, 33, 36, 40, 44, 55, 64, 65, 67, 68, 80, 82] representation of the underlying geometry.”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “Voxel-based GANs directly extend the CNN generators used in 2D settings to 3D [14, 21, 47, 48, 68, 74].”
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images (2019, methodology): “Aiming to separate geometry from texture, visual object networks (VONs) [64] first sample 3D objects from a 3D generative model, render these objects using a hand-crafted differentiable layer to normal, depth and silhouette maps, and finally apply a trained image-to-image translation network.”
Learning Disentangled Representations for Recommendation (2019, background): “sume that each instance is from a concept, while we assume that each instance interacts with objects from different concepts. The majority of the existing efforts are from the ﬁeld of computer vision [28, 15, 20, 30, 53, 14, 10, 39, 19]. Disentangled representation learning on relational data, such ”
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “objects from novel viewpoints. As reasoning in 3D is fundamental for applications in robotics, virtual reality or data augmentation, several recent works consider the task of 3D-aware image synthesis [19,39,76], aiming at photorealistic image generation with explicit control over the camera pose. In”
ClothFlow: A Flow-Based Model for Clothed Person Generation (2019, background): “It is worth noting that existing methods [21, 46, 45] are usually applied to rigid objects where dense correspondences and visibilities are easy to estimate, while the problem we are addressing is more challenging: clothing regions are often highly non-rigid, and there is no clear correspondence bet”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, background): “Earlier works addressing this problem primarily leverage Generative Adversarial Networks (GANs) [3, 4, 12–14, 18, 33, 34, 46, 73].”
Unsupervised Discovery of Disentangled Manifolds in GANs (2020, methodology): “The FineGAN [10] and VON [11] model uses a hierarchy architecture to encode different types of variation into separated latent spaces.”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, background): “Introducing 3D structures into neural networks can improve the quality [37, 41, 44, 48] and controllability of the image generation process [38, 39, 59].”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “Choy et al. (2016), Girdhar et al. (2016), Wu et al. (2015, 2016b) and Zhu et al. (2018) reconstruct and/or generate 3D voxels from a latent representation by directly comparing with available 3D shapes.”
Learning a Neural 3D Texture Space From 2D Exemplars (2019, methodology): “Our comparison will show, how methods tackling space of non-stochastic texture variation [16, 29], unfortunately are not suitable to model stochastic appearance.”