Learning-based view synthesis for light field cameras

By Nima Khademi Kalantari, Tingxian Wang, Ravi Ramamoorthi. 2016

754 citations recorded by the source corpus.

With the introduction of consumer light field cameras, light field imaging has recently become widespread. However, there is an inherent trade-off between the angular and spatial resolution, and thus, these cameras often sparsely sample in either spatial or angular domain. In this paper, we use machine learning to mitigate this trade-off. Specifically, we propose a novel learning-based approach to synthesize new views from a sparse set of input views. We build upon existing view synthesis techniques and break down the process into disparity and color estimation components. We use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. We show the performance of our approach using only four corner sub-aperture views from the light fields captured by the Lytro Illum camera. Experimental results show that our approach synthesizes high-quality images that are superior to the state-of-the-art techniques on a variety of challenging real-world scenes. We believe our method could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning-based Video Motion Magnification (2018, methodology): “Recent techniques demonstrate high-quality results by explicitly shifting pixels using either optical ﬂow [10, 28, 15] or pixel-shifting convolution kernels [18, 19].”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Kalantari et al . proposed a view expansion method for light ﬁeld imaging that uses two sequential convolutional neural networks to model the disparity and color estimation steps of view interpolation and trained these two networks simultaneously [23].”
NeRF: Representing scenes as neural radiance ﬁelds for view synthesis (positive): “Indeed, my own interest in the project that eventually led to NeRF started with depth reconstruction and view interpolation from light ﬁeld cameras [ 23, 69 ], using Lytro cameras from co-author Ren Ng’s startup.”
Multi-content GAN for Few-Shot Font Style Transfer (2017, positive): “CNNs were also used to complete missing regions in images [26] and new stereo and lightfield views [6, 13] given a set of input images.”
Deferred neural rendering (2019, background): “…IBR-approaches learn the blend function using a deep network [Hedman et al. 2018], employ separate networks for color and disparity estimation [Kalantari et al. 2016], explicitly learn view-dependent lighting and shading effects [Thies et al. 2018], or integrate features into a persistent…”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “[28, 23, 7, 62, 31, 30, 63] replace hand-crafted parts of the image-based rendering pipeline with learnable counterparts to improve the robustness.”
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2021, background): “With the advent of deep learning, approaches to light field reconstruction that leverage convolutional neural networks to in-paint or extrapolate light fields from sparse views have been proposed [43, 7, 44], but similarly only support fronto-parallel novel view synthesis.”
IBRNet: Learning Multi-View Image-Based Rendering (2021, methodology): “Recent methods leverage convolutional neural networks (CNNs) to predict volumetric representations stored in voxel grids [20, 26, 27, 42, 59] or multi-plane images (MPIs) [12, 13, 31, 38, 54, 65].”
Light Field Super-Resolution with Zero-Shot Learning (2021, methodology): “Here we adopt a simple structure previously used in [14] as the backbone of AggreNet (details are provided in the supplementary document).”
Neural 3D Video Synthesis from Multi-view Video (2021, neutral): “More recently, [14, 23, 38, 52, 67] learn to fuse and resample pixels from reference views using neural networks.”
Local light field fusion (2019, neutral): “This baseline sub-sumes recent deep learning view synthesis techniques [Flynn et al. 2016; Kalantari et al. 2016], which use a CNN to estimate geometry for each novel view and then backwards warp and blend nearby input images to render the target view.”
Single-View View Synthesis With Multiplane Images (2020, background): “Some approaches predict new views independently for each output view, leading to inconsistency from one view to the next [5, 12].”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, methodology): “CNN-based methods have been applied to synthesizing novel views from sparse light field data [23] or two or more posed images [12, 19, 4].”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, positive): “Although synthesizing novel views in space [41, 61, 45, 25, 20, 7, 16, 81, 71] or time ( i.e ., video frame interpolation) [22, 38, 3, 63] is widely studied, respectively, spatial-temporal synthesis for dynamic scenes is relatively less explored [82, 31].”
DeepView: View Synthesis With Learned Gradient Descent (2019, methodology): “[16] require running a deep network for every desired output view, and so are less suitable for real-time view synthesis.”
Light field video capture using a learning-based hybrid imaging system (2017, positive): “For the disparity CNN, we replace it with the model in (Kalantari et al. 2016), and two other traditional meth-ods (Jeon et al. 2015; Wang et al. 2015).”
Immersive light field video with a layered mesh representation (2020, methodology): “And unlike earlier deep learning methods such as [Flynn et al. 2016; Kalantari et al. 2016] that require running a network to generate each new view, an MPI is self-contained and can be used without a network to render novel views.”
DeepVoxels: Learning Persistent 3D Feature Embeddings (2018, neutral): “Some approaches rely on light ﬁelds [23].”
Light Field Image Processing: An Overview (2017, methodology): “[85], currently constitute the state of the art.”