Video Frame Interpolation via Adaptive Convolution

By Simon Niklaus, Long Mai, Feng Liu. 2017

554 citations recorded by the source corpus.

Video frame interpolation typically involves two steps: motion estimation and pixel synthesis. Such a two-step approach heavily depends on the quality of motion estimation. This paper presents a robust video frame interpolation method that combines these two steps into a single process. Specifically, our method considers pixel synthesis for the interpolated frame as local convolution over two input frames. The convolution kernel captures both the local motion between the input frames and the coefficients for pixel synthesis. Our method employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel. This deep neural network can be directly trained end to end using widely available video data without any difficult-to-obtain ground-truth data like optical flow. Our experiments show that the formulation of video interpolation as a single convolution process allows our method to gracefully handle challenges like occlusion, blur, and abrupt brightness change and enables high-quality video frame interpolation.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “ence can be implemented simultaneously [160], [197], [234], [235].”
Learning-based Video Motion Magnification (2018, methodology): “Recent techniques demonstrated high-quality results by explicitly shifting pixels using either optical flow [10,11,12] or pixelshifting convolution kernels [13,14].”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “[36], we did not use samples that span across video shot boundaries and discarded samples with a lack of texture.”
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement (2018, methodology): “We list the main differencewith flow-basedmethods [9], [36] and kernelbased approaches [12], [28] in Table 1.”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, background): “Video interpolation was the subject of a flurry of interest in the deep learning community a number of years ago [Niklaus et al., 2017, Jiang et al., 2018, Xue et al., 2019, Bao et al., 2019].”
CNN-Based Bi-Directional Motion Compensation for High Efficiency Video Coding (2018, methodology): “[12] presented a video frame interpolation method using the convolutional neural network (CNN).”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, background): “Many other learning strategies including interpolation kernels [43], [44],”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “We provide a visual comparison of video clips with large motions from the Vimeo90K testing set in Figure 7, where SepConv [43] and DAIN [3] produce ghosting artifacts, and CAIN [10] causes missing-parts artifacts.”
Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “…INterpolation), with state-of-the-art end-to-end models including deep voxel ﬂow (DVF) (Liu et al. 2017), adaptive separable convolution (SepConv) (Niklaus, Mai, and Liu 2017b), task-oriented ﬂow (TOFlow) (Xue et al. 2018), SuperSloMo (Jiang et al. 2018), CyclicGen (Liu et al. 2019), and DAIN…”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Most approaches for interpolating between video frames work in 2D image space, by directly predicting kernels that blend two images [51, 52, 53], or by modeling optical flow and warping frames/features [2, 31, 49, 50].”
Spatio-Temporal Filter Adaptive Network for Video Deblurring (2019, methodology): “The standard KPN methods [11, 24, 22] have to predict the filter much larger in size than motion blur for each pixel of the input image, which requires large computational cost and memory.”
Few-shot Video-to-Video Synthesis (2019, background): “This class of networks has a different inductive bias to regular networks and has found use in several tasks including sequence modeling [15], image ﬁltering [23, 59, 49], frame interpolation [38, 37], and neural architecture search [64].”
Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model (2019, methodology): “Similar ideas have been employed in estimating the blur kernels in dynamic deblurring [44, 17] or convolutional kernels in video interpolation [36, 37].”
Depth-Aware Video Frame Interpolation (2019, methodology): “The local interpolation kernels have been shown to be effective for synthesizing a pixel from a large local neighborhood [24, 25].”
A Review on Deep Learning Techniques for Video Prediction (2020, unknown): “SDC-Net [154] 2018 [196], [197] CNN [119], [124] RGB,M RGB X `1, PL SR ? ? X 7 TrIVD-GAN-FP [163] 2020 [142], [162], [166] DVD-GAN [115], [129], [198] RGB RGB X Lhinge [55] R ? ? ? 7 7”
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation (2017, methodology): “Existing methods mainly focus on single-frame video interpolation and have achieved impressive performance for this problem setup [15, 16, 19, 20].”
Burst Denoising with Kernel Prediction Networks (2017, methodology): “Similar ideas have been applied successfully to both video interpolation [18, 19] and video prediction [6, 15, 28, 5], where applying predicted optical flow vectors or filters to the input image data helps prevent the blurry outputs often produced by direct pixel synthesis networks.”
Video Enhancement with Task-Oriented Flow (2017, unknown): “To solve this, some interpolation algorithms [18, 24, 25] estimate occlusion masks and only use pixels that are not occluded in interpolation.”
Context-Aware Synthesis for Video Frame Interpolation (2018, methodology): “Other efforts that use convolutional neural networks for video frame interpolation either had to retrain their synthesis network for a specific t or continue the interpolation recursively in order to achieve this [33, 34].”
Softmax Splatting for Video Frame Interpolation (2020, background): “Research on frame interpolation has seen a recent resurgence, with multiple papers proposing kernel-based [3, 4, 38, 39], ﬂow-based [3, 4, 22, 28, 29, 37, 41, 43, 51], and phase-based [34, 35] approaches.”