GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

By Zekun Hao, Arun Mallya, Serge J. Belongie, Ming-Yu Liu. 2021

134 citations recorded by the source corpus.

We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds such as those created in Minecraft. Our method takes a semantic block world as input, where each block is assigned a semantic label such as dirt, grass, or water. We represent the world as a continuous volumetric function and train our model to render view-consistent photorealistic images for a user-controlled camera. In the absence of paired ground truth real images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training. This stands in contrast to prior work on neural rendering for view synthesis, which requires ground truth images to estimate scene geometry and view-dependent appearance. In addition to camera trajectory, GANcraft allows user control over both scene semantics and output style. Experimental results with comparison to strong baselines show the effectiveness of GANcraft on this novel task of photorealistic 3D block world synthesis. The project website is available at https://nvlabs.github.io/GANcraft/.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Hopefully, with the further advancement of 3D learning-based approaches [126], [127] that aims to render 3D object meshes from a single or few images or even through scene generation approach [128], we will be able to scale up the collection process of large scale 3D datasets.”
Alias-Free Generative Adversarial Networks (2021, background): “They have been used for a variety of applications, including image editing [47, 53, 42, 21, 38, 3], domain translation [66, 36, 58, 41], and video generation [55, 15, 23].”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, background): “3D-Aware Generative Image Synthesis Inspired by the success of neural volume rendering [45] and implicit representations [43, 14], recent work started tackling the problem of 3D-aware image synthesis [7, 57, 49, 26, 25, 76, 8, 51, 58, 67].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “Block-based sparse volume representations overcome some of these issues, but are applicable to mostly empty scenes [19,35] and difﬁcult to generalize across scenes.”
Urban Radiance Fields (2021, methodology): “…this issue, our rendering model includes a spherical radiance (environment) map represented as a coordinate-based neural network, similar to the radiance map used in GANcraft [24] c sky ( d ) = c sky ( d ; θ ) : R 3 → R 3 (10) to provide a direction-dependent background color for those regions.”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[23] Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu.”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, background): “…representations [7, 33, 34, 44] have become a popular representation for various tasks such as 3D reconstruction [1,7,13,14,33,39,42,44,45,48,51,55,57], 3D-aware generative modelling [5, 9, 15, 16, 32, 37, 38, 41, 49, 64], and novel-view synthesis [2, 3, 12, 22, 24, 27, 31, 36, 40, 43, 52, 60, 61].”
GAN-Supervised Dense Visual Alignment (2021, methodology): “[28] Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Conditional [49, 37, 21, 63] and generative models [60, 61, 31] enable a degree”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, background): “Inspired by the success of neural volume rendering [25], recent works started investing in 3D-aware image synthesis [4, 5, 10, 11, 29, 31, 32, 40], which has the advantage of learning 3D generative models directly from images — a more widely accessible resource.”