Video Frame Interpolation via Adaptive Separable Convolution

By Simon Niklaus, Long Mai, Feng Liu. 2017

783 citations recorded by the source corpus.

Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “ence can be implemented simultaneously [160], [197], [234], [235].”
Learning-based Video Motion Magnification (2018, methodology): “Recent techniques demonstrated high-quality results by explicitly shifting pixels using either optical flow [10,11,12] or pixelshifting convolution kernels [13,14].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Standard subsampling and upsampling between convolutions is known to produce artifacts for dense image generation tasks [51], [52].”
Stochastic Adversarial Video Prediction (2018, background): “Standard subsampling and upsam-pling between convolutions is known to produce artifacts for dense image generation tasks [53,54,55].”
A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects (2020, methodology): “The AdaMax algorithm [98] is a variant of Adam that makes the boundary range of the learning rate simpler, and it has been used to train CNN models [101], [102].”
Video Compression through Image Interpolation (2018, background): “Most image interpolation networks build on an encoder-decoder network architecture to move pixels through time [10, 11, 15, 19].”
DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better (2019, methodology): “To alleviate that, we first use a video frame interpolation model [34] to increase the original 240-fps videos to 3840 fps, then perform average pooling over the same time window (but now with more frames).”
A Review on Deep Learning Techniques for Video Prediction (2020, neutral): “SDC-Net [155] 2018 [199], [200] CNN [119], [124] RGB,M RGB X `1, PL SR ? ? X 7 TrIVD-GAN-FP [164] 2020 [142], [163], [167] DVD-GAN [115], [129], [201] RGB RGB X Lhinge [55] R ? ? ? 7 7”
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement (2018, methodology): “We list the main differencewith flow-basedmethods [9], [36] and kernelbased approaches [12], [28] in Table 1.”
VRT: A Video Restoration Transformer (2022, methodology): “[32], [70] and [3] are frame interpolation methods SuperSloMo, SepConv and DAIN, respectively.”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “We provide a visual comparison of video clips with large motions from the Vimeo90K testing set in Figure 7, where SepConv [43] and DAIN [3] produce ghosting artifacts, and CAIN [10] causes missing-parts artifacts.”
Few-shot Video-to-Video Synthesis (2019, background): “This class of networks has a different inductive bias to regular networks and has found use in several tasks including sequence modeling [15], image ﬁltering [23, 59, 49], frame interpolation [38, 37], and neural architecture search [64].”
Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “…INterpolation), with state-of-the-art end-to-end models including deep voxel ﬂow (DVF) (Liu et al. 2017), adaptive separable convolution (SepConv) (Niklaus, Mai, and Liu 2017b), task-oriented ﬂow (TOFlow) (Xue et al. 2018), SuperSloMo (Jiang et al. 2018), CyclicGen (Liu et al. 2019), and DAIN…”
Deep Image Deblurring: A Survey (2022, methodology): “[85] captured 300 video clips at 120 fps and 1080 × 1920 resolution using a GoPro Hero 6 Black camera, and increased the frame rate to 1920 fps by recursively applying frame interpolation [90].”
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation (2017, methodology): “Existing methods mainly focus on single-frame video interpolation and have achieved impressive performance for this problem setup [15, 16, 19, 20].”
Depth-Aware Video Frame Interpolation (2019, methodology): “We show a visual comparison in Figure 7, where the EpicFlow [29], ToFlow [39], SepConv [25] and MEMC-Net [2] methods produce ghosting artifacts on the balls or foot.”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, methodology): “Most approaches for interpolating between video frames work in 2D image space, by directly predicting kernels that blend two images [51, 52, 53], or by modeling optical flow and warping frames/features [2, 31, 49, 50].”