Geometric Image Synthesis

By Hassan Abu Alhaija, Siva Karthik Mustikovela, Andreas Geiger, Carsten Rother. 2018

47 citations recorded by the source corpus.

The task of generating natural images from 3D scenes has been a long standing goal in computer graphics. On the other hand, recent developments in deep neural networks allow for trainable models that can produce natural-looking images with little or no knowledge about the scene structure. While the generated images often consist of realistic looking local patterns, the overall structure of the generated images is often inconsistent. In this work we propose a trainable, geometry-aware image generation method that leverages various types of scene information, including geometry and segmentation, to create realistic looking natural images that match the desired scene structure. Our geometrically-consistent image synthesis method is a deep neural network, called Geometry to Image Synthesis (GIS) framework, which retains the advantages of a trainable method, e.g., differentiability and adaptiveness, but, at the same time, makes a step towards the generalizability, control and quality output of modern graphics rendering engines. We utilize the GIS framework to insert vehicles in outdoor driving scenes, as well as to generate novel views of objects from the Linemod dataset. We qualitatively show that our network is able to generalize beyond the training set to novel scene geometries, object shapes and segmentations. Furthermore, we quantitatively show that the GIS framework can be used to synthesize large amounts of training data which proves beneficial for training instance segmentation models.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, background): “Such biases can be somewhat alleviated through data augmentation, specifically, by harnessing images generated from 3D renderers [39, 4].”
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, methodology): “3D-Aware Image Synthesis: Learning-based novel view synthesis has been intensively investigated in the literature [12,33,51,56–59, 64 , Several recent works exploit generative 3D models for 3D-aware image synthesis [3,27,35,41,63,71].”
Texture Fields: Learning Texture Representations in Function Space (2019, background): “Recent work has shown that image generation methods [3, 11, 19, 20, 24, 30, 47] can be improved by exploiting 3D knowledge about the generated shapes [1, 2, 49].”
Enhancing Photorealism Enhancement (2021, background): “to improve the realism of rendered images [85].”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “While many approaches use additional supervision [2, 10, 87,88,99], we focus on works which are trained on raw image collections like our approach.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “dge about image formation in generative modeling by replicating the inner workings of graphics engines in deep networks. On the synthesis side, geometrydriven generative models using intrinsic images [59,5,57] or the 2.5D image sketch [87] as inputs to image synthesis networks have been shown to del”