SynSin: End-to-End View Synthesis From a Single Image

By Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson. Computer Vision and Pattern Recognition, 2019

545 citations recorded by the source corpus.

View synthesis allows for the generation of new views of a scene given one or more images. This is challenging; it requires comprehensively understanding the 3D scene from images. As a result, current methods typically use multiple images, train on ground-truth depth, or are limited to synthetic data. We propose a novel end-to-end model for this task using a single image at test time; it is trained on real images without any ground-truth 3D information. To this end, we introduce a novel differentiable point cloud renderer that is used to transform a latent 3D point cloud of features into the target view. The projected features are decoded by our refinement network to inpaint missing regions and generate a realistic output image. The 3D component inside of our generative model allows for interpretable manipulation of the latent feature space at test time, e.g. we can animate trajectories from a single image. Additionally, we can generate high resolution images and generalise to other input resolutions. We outperform baselines and prior work on the Matterport, Replica, and RealEstate10K datasets.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/30e911f84f00423625f50ec70b9056ffc3d5b8ca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Free View Synthesis (2020, background): “In a related line of work, new views are synthesized from a single input image [30,44].”
Dynamic View Synthesis from Dynamic Monocular Video (2021, background): “These methods differ in their underlying scene representation, including depth [35, 53], multiplane images [52], or layered depth images [46, 24].”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, background): “Other works [17, 30, 43] train networks for Novel Views Synthesis purely from videos and use point clouds as intermediate 3D representations for conditioning 2D inpainting or generation networks.”
Advances in Neural Rendering (2021, methodology): “Recent neural point rendering methods that adopt this approach include SinSyn and Pulsar [WGSJ20, LZ21].”
A Survey on 3D Gaussian Splatting (2024, unknown): “…strong fitting capability of continuous volumetric radiance fields, essential for high-quality image synthesis, while simultaneously avoiding the computational overhead associated with NeRF based methods ( e.g ., computationally expensive ray-marching, and unnecessary calculations in empty space).”
3D Gaussian Splatting for Real-Time Radiance Field Rendering (2023, background): “There has been recent interest in differentiable point-based rendering techniques [Wiles et al. 2020; Yifan et al. 2019].”
NeX: Real-time View Synthesis with Neural Basis Expansion (2021, background): “When there are only 1-2 input images, the challenge becomes how to infer the ill-constrained 3D geometry and disoccluded regions [38, 23, 43, 47, 3].”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “To alleviate the discontinuity issue, differentiable point-based rendering has been extensively studied to model objects geometry [16, 20, 27, 48, 55].”
Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, background): “Differentiable point-based rendering methods [20, 36, 39, 43, 49, 53, 57] offer great flexibility in representing intricate structures and are thus well-suited for novel view synthesis.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, methodology): “By introducing differentiable 3D projection and using trainable layers that operate in the 3D and 2D feature space, several recent methods [4, 18, 42, 43, 56, 62] are able to model the geometry and appearance of 3D scenes from 2D images.”
Accelerating 3D deep learning with PyTorch3D (2019, background): “Image-based tasks like view synthesis can be improved with 3D representations given only 2D supervision [53, 60, 38].”
NeRF-: Neural Radiance Fields Without Known Camera Parameters (2021, methodology): “One group aims to explicitly reconstruct the surface geometry and the appearance on the surface from the observed sparse views [Chaurasia et al. 2013; Debevec et al. 1996; Hedman et al. 2017; Waechter et al. 2014; Wiles et al. 2020; Zitnick et al. 2004].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Many recent works further push the requirement of the number of input images to a narrow-baseline stereo pair [76, 65, 10] or a single image [77, 69, 50, 70, 30, 61].”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, unknown): “[92] splat latent features from point cloud for novel view synthesis.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “Subsequent efforts generalize MPIs to continuous 3D representations such as NeRF [15] and latent 3D point cloud [55].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Several methods have been proposed that perform novel-view-synthesis from a single image [64, 90, 74, 61, 19].”
Point-NeRF: Point-based Neural Radiance Fields (2022, methodology): “Point clouds have been widely used in rendering, often via rasterization-based point splatting, and even differentiable rasterization modules [25, 51].”