Light field video capture using a learning-based hybrid imaging system

By Tingxian Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, Ravi Ramamoorthi. 2017

134 citations recorded by the source corpus.

Light field cameras have many advantages over traditional cameras, as they allow the user to change various camera settings after capture. However, capturing light fields requires a huge bandwidth to record the data: a modern light field camera can only take three images per second. This prevents current consumer light field cameras from capturing light field videos. Temporal interpolation at such extreme scale (10x, from 3 fps to 30 fps) is infeasible as too much information will be entirely missing between adjacent frames. Instead, we develop a hybrid imaging system, adding another standard video camera to capture the temporal information. Given a 3 fps light field sequence and a standard 30 fps 2D video, our system can then generate a full light field video at 30 fps. We adopt a learning-based approach, which can be decomposed into two steps: spatio-temporal flow estimation and appearance estimation. The flow estimation propagates the angular information from the light field sequence to the 2D video, so we can warp input images to the target view. The appearance estimation then combines these warped images to output the final pixels. The whole process is trained end-to-end using convolutional neural networks. Experimental results demonstrate that our algorithm outperforms current video interpolation methods, enabling consumer light field videography, and making applications such as refocusing and parallax view generation achievable on videos for the first time.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning-based Video Motion Magnification (2018, methodology): “Recent techniques demonstrate high-quality results by explicitly shifting pixels using either optical ﬂow [10, 28, 15] or pixel-shifting convolution kernels [18, 19].”
Light Field Image Processing: An Overview (2017, unknown): “[73] Learning-based Temporal (10×) – – – – – –”
Immersive light field video with a layered mesh representation (2020, methodology): “This has been addressed by conventional video compression methods [Avramelos et al. 2019] or deep learning methods for super-resolution [Wang et al. 2017], but neither of these approaches produces a viewing volume sufficient for our needs.”
Video Enhancement with Task-Oriented Flow (2017, unknown): “Such approaches have been applied to video interpolation (Liu et al 2017), light-ﬁeld interpolation (Wang et al 2017), novel view synthesis (Zhou et al 2016), eye gaze manipulation (Ganin et al 2016), object detection (Zhu et al 2017), denoising (Wen et al 2017), and super-resolution (Caballero et…”
End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging (2018, background): “2017] and to produce light field video clips from a hybrid camera [Wang et al. 2017].”