Interactive 3D Modeling with a Generative Adversarial Network

By Jerry Liu, Fisher Yu, Thomas A. Funkhouser. 2017

95 citations recorded by the source corpus.

We propose the idea of using a generative adversarial network (GAN) to assist users in designing real-world shapes with a simple interface. Users edit a voxel grid with a Minecraft-like interface. Yet they can execute a SNAP command at any time, which transforms their rough model into a desired shape that is both similar and realistic. They can edit and snap until they are satisfied with the result. The advantage of this approach is to assist novice users to create 3D models characteristic of the training data by only specifying rough edits. Our key contribution is to create a suitable projection operator around a 3D-GAN that maps an arbitrary 3D voxel input to a latent vector in the shape manifold of the generator that is both similar in shape to the input but also realistic. Experiments show our method is promising for computer-assisted interactive modeling.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, background): “been combined with generative adversarial approaches [8] to generate novel 3D objects [46, 41, 20], but only at a limited resolution.”
Multi-chart generative surface modeling (2018, background): “Another type of input can be supplied by the user: [23] suggested a system that is based on 3D GANs and user interaction that generates 3D models.”
Editing Conditional Radiance Fields (2021, background): “There has also been work to edit the appearance of a 3D scene, e.g., via transferring multi-channel edits to other views [24], scribble-based material transfer [4], editing 3D shapes in a voxel representation [37], and relighting a scene with a paint brush interface [53].”
Image2Mesh: A Learning Framework for Single Image 3D Reconstruction (2017, methodology): “s, pooling, and other techniques that have been successfully applied to 2D images for the learning process. Volumetric autoencoders [6,24] and generative adversarial networks (GAN) have been proposed [7,15,32] to learn of the probabilistic latent space of object shapes. It has been used for object c”
Improved Adversarial Systems for 3D Object Generation and Reconstruction (2017, background): “Several of these shape models have been shown in practical applications such as interactive editing [20] and reconstructing room geometry from a single image [4, 24, 13].”
MineRL: A Large-Scale Dataset of Minecraft Demonstrations (2019, background): “This inability is further evidenced by the large-body of work developed on Minecraft-like domains which speciﬁcally captures restricted subsets of the features of Minecraft [ Salge et al. , 2014 ] , [ Andreas et al. , 2017 ] , [ Liu et al. , 2017 ] .”
Generative Adversarial Networks–Enabled Human–Artificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends (2021, methodology): “The system proposed by Liu et al. (2017) presents an interactive 3D modeling tool, assisting users in designing real-5 world shapes using a simple voxel-based interface.”