Single-View View Synthesis With Multiplane Images

By Richard Tucker, Noah Snavely. 2020

405 citations recorded by the source corpus.

A recent strand of work in view synthesis uses deep learning to generate multiplane images—a camera-centric, layered 3D representation—given two or more input images at known viewpoints. We apply this representation to single-view view synthesis, a problem which is more challenging but has potentially much wider application. Our method learns to predict a multiplane image directly from a single image input, and we introduce scale-invariant view synthesis for supervision, enabling us to train on online video. We show this approach is applicable to several different datasets, that it additionally generates reasonable depth maps, and that it learns to fill in content behind the edges of foreground objects in background layers.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, background): “Several prior works have explored ways to leverage depth information for view synthesis [24,27] and NeRF training [9, 10, 16, 18, 30].”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, neutral): “In contrast, approaches using multi-plane images (MPIs) [11,28,55,56,64,78] represent the scene as a stack of images or feature maps.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, positive): “These networks leverage large collections of posed multiview data across many scenes [8, 11, 19, 21, 44, 55, 57, 60, 67, 71].”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “Among few-shot NVS methods, some rely on test-time optimization [70, 29] or meta learning [67, 77], while others lift input observations via encoders [80, 45, 94, 89, 79, 11, 81] and predict novel views in a feed-forward fashion.”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Many recent works further push the requirement of the number of input images to a narrow-baseline stereo pair [76, 65, 10] or a single image [77, 69, 50, 70, 30, 61].”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, background): “On the other hand, locally conditioned models [47, 63, 55, 52, 53] aim to use local image features directly for scene reconstruction and show greater cross-domain generalization capabilities, though are generally limited to close-by view reconstructions.”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, background): “More recently, the problem of predicting novel views from a single image has been explored [44, 47, 36, 5].”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, methodology): “Recent approaches employ 2D/3D Convolutional Neural Network (CNNs) to estimate multiplane images (MPIs) [12, 26, 36, 51, 56, 71] for forward-facing captures; estimate voxel grid [17, 32, 48] for inward-facing captures.”
NeRF-: Neural Radiance Fields Without Known Camera Parameters (2021, background): “…of this kind, such as Soft3D [Penner and Zhang 2017], Multi-Plane Images (MPI) [Choi et al. 2019; Flynn et al. 2019; Milden-hall et al. 2019; Tucker and Snavely 2020; Zhou et al. 2018], Scene Representation Networks (SRN) [Sitzmann et al. 2019], Occupancy Networks [Mescheder et al. 2019;…”
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D (2020, methodology): “These architectures, as well as ours, use data structures similar to “multi-plane” images from the machine learning graphics community [34,32,38,20].”
Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, background): “Note that while conducting comparisons with other view synthesis models (such as [42, 34, 19, 28, 12, 43, 40]) and few-shot recognition models (such as [2, 16, 6]) are interesting, this is not the main focus of our paper.”
Dynamic View Synthesis from Dynamic Monocular Video (2021, background): “These methods differ in their underlying scene representation, including depth [35, 53], multiplane images [52], or layered depth images [46, 24].”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “Early researches [3], [4], [5], [6], [7], [8], [9], [10] focused on training regression-based models to synthesize novel views from sparse or single input.”
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior (2022, background): “meshes [30, 31], Multi-Plane Images [41, 59], layered depth [42] etc.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, methodology): “Various representations have been explored in this direction, e.g . depth [15, 61, 42, 43], layered depth [56, 46], multi-plane images [71, 51, 55], volume density [27, 31], and mesh sheets [20].”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “One example of a mixture-based representation are Multi-Plane Images (MPIs) [Srinivasan et al. 2019; Tucker and Snavely 2020; Zhou et al. 2018].”
NeX: Real-time View Synthesis with Neural Basis Expansion (2021, background): “Subsequent work extends MPI to support multiple input photos [6, 21, 17] or even infers an MPI from a single image [43].”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “For example, [48] predicts MPIs for view synthesis from single image without requiring ground truth 3D.”
Advances in Neural Rendering (2021, methodology): “…to mitigating the memory challenges of dense voxel grids include using object-speciﬁc shape templates [KTEM18], multi-plane [ZTF ∗ 18, MSOC ∗ 19, FBD ∗ 19, TS20, WPYS21] or multi-sphere [BFO ∗ 20, ALG ∗ 20] images, which all aim at representing the voxel grid using a sparse approximation.”