Transformation-Grounded Image Generation Network for Novel 3D View Synthesis

By Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, Alexander C. Berg. 2017

294 citations recorded by the source corpus.

We present a transformation-grounded image generation network for novel 3D view synthesis from a single image. Our approach first explicitly infers the parts of the geometry visible both in the input and novel views and then casts the remaining synthesis problem as image completion. Specifically, we both predict a flow to move the pixels from the input to the novel view along with a novel visibility map that helps deal with occulsion/disocculsion. Next, conditioned on those intermediate results, we hallucinate (infer) parts of the object invisible in the input image. In addition to the new network structure, training with a combination of adversarial and perceptual loss results in a reduction in common artifacts of novel view synthesis such as distortions and holes, while successfully generating high frequency details and preserving visual aspects of the input image. We evaluate our approach on a wide range of synthetic and real examples. Both qualitative and quantitative results show our method achieves significantly better results compared to existing methods.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Generative Image Inpainting with Contextual Attention (2018, methodology): “It has many applications in photo editing, image-based rendering and computational photography [3, 23, 28, 29, 34, 39].”
Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks (2017, background): “View based representations have demonstrated strong potential for both shape recognition and synthesis [23, 18, 5, 25, 16, 13, 29].”
Pluralistic Image Completion (2019, methodology): “These have been applied to conditional image generation tasks, such as image translation [15], synthetic to realistic [44], future prediction [27], and 3D models [28].”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, background): “Related to the problem of 3D texture inference are view-synthesis approaches that predict novel views from a single image [73, 43] or multiple images [56].”
Novel View Synthesis with Diffusion Models (2022, background): “…et al., 2019) and Neural Radiance Fields (NeRF) (Mildenhall et al., 2020), state-of-the-art approaches to novel view synthesis were typically built on generative models (Sun et al., 2018) or more classical techniques on interpolation or disparity estimation (Park et al., 2017; Zhou et al., 2018).”
Flow-Grounded Spatial-Temporal Video Prediction from Still Images (2018, background): “The similar idea is also applied in the task of novel view synthesis [34].”
Texture Fields: Learning Texture Representations in Function Space (2019, methodology): “In [27], point clouds are reconstructed from a set of novel views using multi-view stereo algorithms.”
Deferred neural rendering (2019, unknown): “In addition we provide a background image to the neural renderer, to output the final reenactment result. warping [Park et al. 2017; Zhou et al. 2016], or a layered scene representation [Tulsiani et al. 2018; Zhou et al. 2018].”
Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation (2019, methodology): “It can be widely applied in many tasks [Barnes et al., 2009a; Newson et al., 2014; Park et al., 2017; Simakov et al., 2008], such as photo editing, image-based rendering, computational photography, etc. Till now, there have been many methods proposed for generating desirable ∗Corresponding…”
An Unsupervised Learning Model for Deformable Medical Image Registration (2018, methodology): “The layer has since been used for dense spatial transformations as well [34, 46].”
PEPSI : Fast Image Inpainting With Parallel Decoding Network (2019, unknown): “Image inpainting techniques have been widely researched [1–3, 5, 6, 10, 13, 16, 17, 19, 21, 25, 26, 28] for removing an unwanted object or synthesizing missing parts of an image in various applications such as photo editing, image-based rendering, and computational photography [15, 20, 28].”
GAN-Supervised Dense Visual Alignment (2021, background): “, classification) and applications such as robust filter learning [16, 36], view synthesis [26, 63, 90] and 3D representation learning [38, 83, 88].”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, background): “The goal is to estimate one or multiple depth maps, which can be from the same viewpoint as the input [2], [3], [4], [5], or from a new arbitrary viewpoint [6], [7], [8], [9], [10].”
Supplementary: Monocular Neural Image Based Rendering with Continuous View Control (2019, methodology): “The proposed network achieves the best structural similarity results among all previous flow [7, 3] or pixel [6] based methods on both ShapeNet and KITTI datasets.”
DeepVoxels: Learning Persistent 3D Feature Embeddings (2018, background): “Some techniques predict lookup tables into a set of reference views [39, 63] or predict weights to blend multi-view images into novel views [11].”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, background): “Introducing 3D structures into neural networks can improve the quality [37, 41, 44, 48] and controllability of the image generation process [38, 39, 59].”
Multi-content GAN for Few-Shot Font Style Transfer (2017, unknown): “For example, given a single-view image, encoder-decoder architectures have been proposed to halluci-nate novel views of faces [15, 31, 4], bodies [35], and other rigid objects [37, 25].”