Context-Aware Synthesis for Video Frame Interpolation

By Simon Niklaus, Feng Liu. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

440 citations recorded by the source corpus.

Video frame interpolation algorithms typically estimate optical flow or its variations and then use it to guide the synthesis of an intermediate frame between two consecutive original frames. To handle challenges like occlusion, bidirectional flow between the two input frames is often estimated and used to warp and blend the input frames. However, how to effectively blend the two warped frames still remains a challenging problem. This paper presents a context-aware synthesis approach that warps not only the input frames but also their pixel-wise contextual information and uses them to interpolate a high-quality intermediate frame. Specifically, we first use a pre-trained neural network to extract per-pixel contextual information for input frames. We then employ a state-of-the-art optical flow algorithm to estimate bidirectional flow between them and pre-warp both input frames and their context maps. Finally, unlike common approaches that blend the pre-warped frames, our method feeds them and their context maps to a video frame synthesis neural network to produce the interpolated frame in a context-aware fashion. Our neural network is fully convolutional and is trained end to end. Our experiments show that our method can handle challenging scenarios such as occlusion and large motion and outperforms representative state-of-the-art approaches.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/65fadccad0fc743876a259c2b779622636c2ffde

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement (2018, methodology): “We evaluate the proposed MEMC-Net and MEMC-Net* against the kernel-based method (SepConv [12]), flowbased algorithms (DVF [36], ToFlow [9], and CtxSyn [27]), and a direct interpolation approach (MIND [7]).”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “Video synthesis involves a wide range of tasks including video-to-video translation [73], image animation [64, 65], frame interpolation [52, 4, 53], and video prediction.”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “Research on frame interpolation has seen a recent resurgence, with multiple papers proposing kernel-based [3, 4, 38, 39], ﬂow-based [3, 4, 22, 28, 29, 37, 41, 43, 51], and phase-based [34, 35] approaches.”
Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “Video frame interpolation approaches (Jiang et al. 2018; Liu et al. 2017; Niklaus and Liu 2018; Xue et al. 2018) also follow this trend and typically work as follows: 1) estimating optical ﬂow between two consecutive frames, 2) constructing initial estimates of intermediate frames by image warping…”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “Recently, ﬂow-based VFI algorithms have oﬀered a framework to address these challenges and achieved impressive re-sults [30,22,40,62,3,61,28].”
Portrait shadow manipulation (2020, methodology): “…computation to happen at different layers and different spatial scales instead of conflating layers and spatial scales (as U-Nets do) GridNet produces more accurate predictions as has been successfully applied to a number of image synthesis tasks (Niklaus and Liu 2018; Niklaus et al. 2019).”
Video Enhancement with Task-Oriented Flow (2017, positive): “Inspired by the success of deep learning, researchers have directly modeled enhancement tasks as regression problems without representing motions, and have designed deep networks for frame interpolation (Mathieu et al 2016; Niklaus et al 2017a; Jiang et al 2017; Niklaus and Liu 2018), super-resoluti”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, background): “[45], context maps [46], and incorporation of depth information [6] can effectively improve the interpolation quality.”