Pushing the Boundaries of View Extrapolation With Multiplane Images

By Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, Noah Snavely. Computer Vision and Pattern Recognition, 2019

355 citations recorded by the source corpus.

We explore the problem of view synthesis from a narrow baseline pair of images, and focus on generating high-quality view extrapolations with plausible disocclusions. Our method builds upon prior work in predicting a multiplane image (MPI), which represents scene content as a set of RGBA planes within a reference view frustum and renders novel views by projecting this content into the target viewpoints. We present a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to 4 times the lateral viewpoint movement allowed by prior work. Our method ameliorates two specific issues that limit the range of views renderable by prior methods: 1) We expand the range of novel views that can be rendered without depth discretization artifacts by using a 3D convolutional network architecture along with a randomized-resolution training procedure to allow our model to predict MPIs with increased disparity sampling frequency. 2) We reduce the repeated texture artifacts seen in disocclusions by enforcing a constraint that the appearance of hidden content at any depth must be drawn from visible content at or behind that depth.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4b4bae07638f988556ec6b8a8f46b514a02841e5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

NeX: Real-time View Synthesis with Neural Basis Expansion (2021, methodology): “[37] predicts an MPI using a two-step process that combines 3D CNNs for MPI prediction and a 2D flow field for warping RGB values from an intermediate rendering.”
Dense Depth Priors for Neural Radiance Fields from Sparse Input Views (2021, neutral): “More recently, these mesh-based geometry models have been replaced by volumetric scene representations such as voxel grids [18] or multiplane images [11, 20, 24, 29].”
Urban Radiance Fields (2021, background): “Others re-construct volumetric representations, such as voxels [55], octrees [65], or multi-plane images [18, 58, 76].”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Yet another promising 3D representation is multiplane images (MPIs), that have been shown to model complex scene appearance [9, 10, 15, 20, 45, 69].”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “…mesh reconstruction by using volumetric representations of geometry and appearance, such as voxel grids [Lombardi et al. 2019; Penner and Zhang 2017; Szeliski and Golland 1999; Vo-giatzis et al. 2007] or multiplane images [Srinivasan et al. 2019; Wizadwongsa et al. 2021; Zhou et al. 2018].”
SynSin: End-to-End View Synthesis From a Single Image (2019, background): “Recent methods [56, 68, 74] can be extremely effective even for complex real-world scenes.”
PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, background): “Volumebased representations such as voxel grids [40, 17, 23, 13, 53, 42] and multi-plane images (MPIs) [47, 33, 62, 46, 27] are a popular alternative to mesh representations due to their topology-free nature: gradient-based optimization is therefore straightforward, while rendering can still be real”
Single-View View Synthesis With Multiplane Images (2020, background): “For instance, multiplane images (MPIs), originally devised for stereo matching problems [30], have recently found success in both learned view interpolation and extrapolation from multiple input images [37, 27, 4, 21].”
Neural Scene Graphs for Dynamic Scenes (2020, methodology): “Existing methods have proposed to learn features on discrete geometric primitives, such as points [2, 27], multi-planes [10, 17, 21, 35, 46], meshes [6, 39] higherorder primitives like voxel grids [16, 31, 44], or implicitly represent features [15, 20, 24, 26, 43, 22] using functions F : R(3) → R th”
Neural 3D Video Synthesis from Multi-view Video (2021, neutral): “More recently, [14, 23, 38, 52, 67] learn to fuse and resample pixels from reference views using neural networks.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, neutral): “DietNeRF [10] penalizes a semantic consistency loss by minimizing distance between CLIP [33] them, a number of methods rely on layered representations such as layered depth images [40, 48, 41] and multi-plane images (MPIs) [43, 47, 17, 8].”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, methodology): “Recent approaches employ 2D/3D Convolutional Neural Network (CNNs) to estimate multiplane images (MPIs) [12, 26, 36, 51, 56, 71] for forward-facing captures; estimate voxel grid [17, 32, 48] for inward-facing captures.”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, neutral): “Another popular class of view synthesis methods uses discretized volumetric representations such as voxel grids [25, 36, 37, 39] or multiplane images [12, 33, 40, 48].”
Immersive light field video with a layered mesh representation (2020, methodology): “There are some visual phenomenon which are not accurately represented by MPIs or MSIs, such as curved reflectors.”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, methodology): “Reference Frame Zoom-in StereoMag [72]PB-MPI [52] LLFF [39] XView [4] Ours”
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “3D-Aware Image Synthesis: Learning-based novel view synthesis has been intensively investigated in the literature [12,33,51,56–59, 64 , Several recent works exploit generative 3D models for 3D-aware image synthesis [3,27,35,41,63,71].”
Editing Conditional Radiance Fields (2021, background): “Prior work reasons over an explicit [11, 15, 22, 72] or discrete volumetric [20, 33, 36, 40, 44, 55, 64, 65, 67, 68, 80, 82] representation of the underlying geometry.”