Depth-Aware Video Frame Interpolation

By Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang. Computer Vision and Pattern Recognition, 2019

579 citations recorded by the source corpus.

Video frame interpolation aims to synthesize nonexistent frames in-between the original frames. While significant advances have been made from the recent deep convolutional neural networks, the quality of interpolation is often reduced due to large object motion or occlusion. In this work, we propose a video frame interpolation method which explicitly detects the occlusion by exploring the depth information. Specifically, we develop a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones. In addition, we learn hierarchical features to gather contextual information from neighboring pixels. The proposed model then warps the input frames, depth maps, and contextual features based on the optical flow and local interpolation kernels for synthesizing the output frame. Our model is compact, efficient, and fully differentiable. Quantitative and qualitative results demonstrate that the proposed model performs favorably against state-of-the-art frame interpolation methods on a wide variety of datasets. The source code and pre-trained model are available at https://github.com/baowenbo/DAIN.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6c8a56ae495e5c8871061d1cd0f863d174f5e2ce

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, background): “Video interpolation was the subject of a flurry of interest in the deep learning community a number of years ago [Niklaus et al., 2017, Jiang et al., 2018, Xue et al., 2019, Bao et al., 2019].”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Most approaches for interpolating between video frames work in 2D image space, by directly predicting kernels that blend two images [51, 52, 53], or by modeling optical flow and warping frames/features [2, 31, 49, 50].”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “Research on frame interpolation has seen a recent resurgence, with multiple papers proposing kernel-based [3, 4, 38, 39], ﬂow-based [3, 4, 22, 28, 29, 37, 41, 43, 51], and phase-based [34, 35] approaches.”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “Bao et al. [3] propose DAIN to estimate the intermediate ﬂow as a weighted combination of bidirectional ﬂow.”
Deep Contextual Video Compression (2021, background): “Other video tasks show that utilizing temporal information as condition is helpful [27, 28].”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “Video synthesis involves a wide range of tasks including video-to-video translation [79], image animation [69, 70], frame interpolation [56, 4], and video prediction.”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, unknown): “, video frame interpolation) [3, 22, 39, 62] is widely studied, respectively, spatial-temporal synthesis for dynamic scenes is relatively less explored [32, 79].”
VRT: A Video Restoration Transformer (2022, methodology): “[32], [70] and [3] are frame interpolation methods SuperSloMo, SepConv and DAIN, respectively.”
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis (2021, background): “While some works solely focus on transferring a predefined holistic motion between objects [73] or interpolating motion between a starting and end frame [54, 79, 43, 4, 55], the most commonly addressed problem is video prediction.”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, background): “recent works [6], [9], [57] have shown that multi-stage training and pre-trained models can be beneficial in this scenario.”
Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “2 Comparison with the state-of-the-art We compare our framework, named CAIN (Channel Attention for frame INterpolation), with state-of-the-art end-to-end models including deep voxel ﬂow (DVF) (Liu et al. 2017), adaptive separable convolution (SepConv) (Niklaus, Mai, and Liu 2017b), task-oriented ﬂow”
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 (2021, unknown): “3 [7] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.”