DeepView: View Synthesis With Learned Gradient Descent

By John Flynn, Michael Broxton, Paul E. Debevec, Matthew DuVall, Graham Fyffe, Ryan S. Overbeck, Noah Snavely, Richard Tucker. Computer Vision and Pattern Recognition, 2019

487 citations recorded by the source corpus.

We present a novel approach to view synthesis using multiplane images (MPIs). Building on recent advances in learned gradient descent, our algorithm generates an MPI from a set of sparse camera viewpoints. The resulting method incorporates occlusion reasoning, improving performance on challenging scene features such as object boundaries, lighting reflections, thin structures, and scenes with high depth complexity. We show that our method achieves high-quality, state-of-the-art results on two datasets: the Kalantari light field dataset, and a new camera array dataset, Spaces, which we make publicly available.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0c0eb3040a32484fcb6270841e14b78f284e7570

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

IBRNet: Learning Multi-View Image-Based Rendering (2021, methodology): “Recent methods leverage convolutional neural networks (CNNs) to predict volumetric representations stored in voxel grids [20, 26, 27, 42, 59] or multi-plane images (MPIs) [12, 13, 31, 38, 54, 65].”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “[18] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.”
Dynamic View Synthesis from Dynamic Monocular Video (2021, background): “Prior work addresses this via local warps [9], operating in the gradient domain [23], soft 3D reconstruction [41], and learning-based approaches [21, 15, 14, 20, 44].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Multi-plane images (MPIs) have been popular in the past few years as a geometric representation [76, 43, 17, 32], allowing for compelling novel view synthesis quality.”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Neural scene representations often use an explicit 3D proxy geometry, such as multi-plane [15, 36, 62] or multi-sphere [2, 4] images or a voxel grid of features [31, 52].”
Neural Sparse Voxel Fields (2020, unknown): “, 2019a, 2020), multi-plane images (Zhou et al., 2018; Flynn et al., 2019; Mildenhall et al., 2019), point clouds (Meshry et al.”
NeX: Real-time View Synthesis with Neural Basis Expansion (2021, methodology): “Subsequent work extends MPI to support multiple input photos [6, 21, 17] or even infers an MPI from a single image [43].”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, unknown): “3.2.”
Advances in Neural Rendering (2021, methodology): “…to mitigating the memory challenges of dense voxel grids include using object-speciﬁc shape templates [KTEM18], multi-plane [ZTF ∗ 18, MSOC ∗ 19, FBD ∗ 19, TS20, WPYS21] or multi-sphere [BFO ∗ 20, ALG ∗ 20] images, which all aim at representing the voxel grid using a sparse approximation.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, unknown): “[26] combine multi-plane images with learned gradient descent.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “Storage can be reduced by interpolating intermediate images with optical flow [5], representing the light field as a neural network [1], or by reconstructing a Multi-Plane Image (MPI) representation of the scene [15,32,37,47,54].”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “Similarly, multi-plane image [Flynn et al. 2019; Mildenhall et al. 2019; Wizadwongsa et al. 2021; Zhou et al. 2018] or multi-sphere image [Attal et al.”
Immersive light field video with a layered mesh representation (2020, methodology): “Our analysis is similar to conditions for the spacing of MPI planes proposed in [Flynn et al. 2019; Zhou et al. 2018] and further refined using Fourier analysis in [Mildenhall et al. 2019].”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, unknown): “In contrast, approaches using multi-plane images (MPIs) [11,28,55,56,64,78] represent the scene as a stack of images or feature maps.”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, background): “Another popular class of methods uses discretized volumetric representations such as voxel grids [27, 41, 42, 45] or multiplane images [13, 37, 46, 56].”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, unknown): “Recent works have used volumetric representations such as voxel grids [18] or multiplane images [9,21,35,39,46], which are better suited for gradient-based optimization than meshes.”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, unknown): “Although synthesizing novel views in space [41, 61, 45, 25, 20, 7, 16, 81, 71] or time ( i.e ., video frame interpolation) [22, 38, 3, 63] is widely studied, respectively, spatial-temporal synthesis for dynamic scenes is relatively less explored [82, 31].”