MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement

By Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018

363 citations recorded by the source corpus.

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net architecture can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/d833c48334e906537f21757b6f9fa44da66f6c76

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Depth-Aware Video Frame Interpolation (2019, methodology): “We use the adaptive warping layer [2] to warp the input frames, contextual features, and depth maps based on the estimated flows and local interpolation kernels.”
Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “…and segmentation (Cheng et al. 2017; Zhu et al. 2018; 2017), video prediction (Li et al. 2018; Reda et al. 2018), and others (Chen et al. 2017; Lai et al. 2018)—frequently depend on the accurate optical ﬂow estimation since it leads to the success of the target tasks in terms of quantitative…”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “To provide a quantitative comparison of multiple frame interpolation, we further extract every fourth frame of videos from HD benchmark [4] and use them to interpolate other frames.”
Softmax Splatting for Video Frame Interpolation (2020, background): “Research on frame interpolation has seen a recent resurgence, with multiple papers proposing kernel-based [3, 4, 38, 39], ﬂow-based [3, 4, 22, 28, 29, 37, 41, 43, 51], and phase-based [34, 35] approaches.”