Deep blending for free-viewpoint image-based rendering

By Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, Gabriel J. Brostow. ACM Transactions on Graphics, 2018

870 citations recorded by the source corpus.

Free-viewpoint image-based rendering (IBR) is a standing challenge. IBR methods combine warped versions of input photos to synthesize a novel view. The image quality of this combination is directly affected by geometric inaccuracies of multi-view stereo (MVS) reconstruction and by view- and image-dependent effects that produce artifacts when contributions from different input views are blended. We present a new deep learning approach to blending for IBR, in which we use held-out real image data to learn blending weights to combine input photo contributions. Our Deep Blending method requires us to address several challenges to achieve our goal of interactive free-viewpoint IBR navigation. We first need to provide sufficiently accurate geometry so the Convolutional Neural Network (CNN) can succeed in finding correct blending weights. We do this by combining two different MVS reconstructions with complementary accuracy vs. completeness tradeoffs. To tightly integrate learning in an interactive IBR system, we need to adapt our rendering algorithm to produce a fixed number of input layers that can then be blended by the CNN. We generate training data with a variety of captured scenes, using each input photo as ground truth in a held-out approach. We also design the network architecture and the training loss to provide high quality novel view synthesis, while reducing temporal flickering artifacts. Our results demonstrate free-viewpoint IBR in a wide variety of scenes, clearly surpassing previous methods in visual quality, especially when moving far from the input cameras.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/30453b89e7d83f8142e5010a0998175bffc56025

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

3D Photography Using Context-Aware Layered Depth Inpainting (2020, methodology): “Recent advances include learning-based blending [19], soft 3D reconstruction [45], handling reflection [49, 26], relighting [63], and reconstructing mirror and glass surfaces [59].”
Enhancing Photorealism Enhancement (2021, background): “real imagery of a scene for rendering novel views [36], [37], [38], [39], [40], [41], [42], [43], [44].”
Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis (2023, positive): “We evaluate our compression and rendering method on the Mip-Nerf360[2] indoor and outdoor scenes, two scenes from the Tanks&Temples[14] and Deep Blending [9] dataset, and NeRF-Synthetic[16] .”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “PSNR curve of Scaffold -GS and 3D-GS [22] across diverse datasets [4, 17, 49].”
Deferred neural rendering (2019, methodology): “Recent learning-based IBR-approaches learn the blend function using a deep network [Hedman et al. 2018], employ separate networks for color and disparity estimation [Kalantari et al.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, methodology): “While early methods applied view-dependent texturing to a 3D mesh [7, 12, 13], later methods incorporated convolutional neural networks as a post-processing step to improve quality [20, 31, 44].”
A Survey on 3D Gaussian Splatting (2024, neutral): “All the learnable parameters are optimized by stochastic gradient descent using the ℓ 1 and D-SSIM loss functions: where 𝜆 ∈ [ 0 , 1 ] is a weighting factor.”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, neutral): “Although synthesizing novel views in space [41, 61, 45, 25, 20, 7, 16, 81, 71] or time ( i.e ., video frame interpolation) [22, 38, 3, 63] is widely studied, respectively, spatial-temporal synthesis for dynamic scenes is relatively less explored [82, 31].”
Single-View View Synthesis With Multiplane Images (2020, methodology): “A number of classical approaches to this problem have been explored [10, 16], including methods that involve estimation of local geometric proxies [1, 11, 23, 38].”
Free View Synthesis (2020, methodology): “[15] use a deep convolutional network to blend source images that have been warped into the target view.”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, methodology): “Finally, a very recent IBRNet [68] learns to copy existing colors from known views in an IBR fashion [24, 5], whereas our method can hallucinate new colors, which is crucial since we leverage far fewer source views (at most 9).”
Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields (2021, methodology): “We additionally compare with the publicly available SIBR implementations [7] of Deep Blending [15] and Point-Based Neural Rendering [23], two real-time IBR-based view synthesis approaches that also depend on an external proxy geometry.”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, background): “[66] averages features over multiple views and [61] learns to interpolate between views in an IBR fashion [18, 2] which prevents inpainting unseen areas.”
SynSin: End-to-End View Synthesis From a Single Image (2019, methodology): “[1, 9, 23, 36, 38, 41] use a DNN to improve view synthesis from a set of noisy, incomplete, or inconsistent depth maps.”
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, methodology): “, a reconstructed triangle mesh) to reproject and combine colors from the input images, using either a heuristic [6] or learned [22, 42, 43] blending function.”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “[27, 22, 7, 57] replace hand-crafted parts of the image-based rendering pipeline with learnable counterparts to improve the robustness.”