3D Photography Using Context-Aware Layered Depth Inpainting

By Meng-Li Shih, Shih-Yang Su, Johannes Peter Kopf, Jia-Bin Huang. Computer Vision and Pattern Recognition, 2020

342 citations recorded by the source corpus.

We propose a method for converting a single RGB-D input image into a 3D photo, i.e., a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that iteratively synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show less artifacts when compared with the state-of-the-arts.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/909d1109959ea980745a43a717a50a1478bce1fb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “5) 3D-Photo [42], depthbased image warping and inpainting method.”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, methodology): “Since we are not aware of prior monocular space-time view interpolation methods, we use two state-of-the-art view synthesis baselines [64, 81] to synthesize images at the testing camera viewpoints followed by 2D frame interpolation [52] to render intermediate times, as well as NeRF evaluated directl”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Many recent works further push the requirement of the number of input images to a narrow-baseline stereo pair [76, 65, 10] or a single image [77, 69, 50, 70, 30, 61].”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, background): “Several prior works have explored ways to leverage depth information for view synthesis [24,27] and NeRF training [9, 10, 16, 18, 30].”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, methodology): “Since this method [59] takes inverse depth in pretraining, thus the manual transformation of our depth into its inverse form will cause accuracy loss.”
NeRF-: Neural Radiance Fields Without Known Camera Parameters (2021, neutral): “Apart from novel views synthesis using multiple images, there are also learning-based approaches [Niklaus et al. 2019; Shih et al. 2020; Tucker and Snavely 2020; Wiles et al. 2020; Wu et al. 2020; Zhou et al. 2016], which allow for single-image novel view synthesis at inference time by learning a…”
Consistent video depth estimation (2020, background): “2018], novel view synthesis [Hedman et al. 2017; Hedman and Kopf 2018; Hedman et al. 2018; Shih et al. 2020], and occlusionaware augmented reality [Holynski and Kopf 2018].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Several methods have been proposed that perform novel-viewsynthesis from a single image [64, 90, 74, 61, 19].”
Dynamic View Synthesis from Dynamic Monocular Video (2021, background): “These methods differ in their underlying scene representation, including depth [39, 57], multiplane images [56], or layered depth images [50, 25].”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “We simplify the 3D-GS optimization process by deprecating the densi-fication, splitting, and opacity reset tricks [69], and reduce the overall optimization time into 2,000 iterations, which is considerably faster than the original 3D-GS training.”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “Layered depth images [11, 45, 47, 57] work for sparse input views but rely on depth maps or estimated depth with sacrificed quality.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “Various representations have been explored in this direction, e.g . depth [15, 61, 42, 43], layered depth [56, 46], multi-plane images [71, 51, 55], volume density [27, 31], and mesh sheets [20].”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, background): “More recently, the problem of predicting novel views from a single image has been explored [44, 47, 36, 5].”