Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

By Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Eric Learned-Miller, Jan Kautz. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

909 citations recorded by the source corpus.

Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences. While most existing methods focus on single-frame interpolation, we propose an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. We start by computing bi-directional optical flow between the input images using a U-Net architecture. These flows are then linearly combined at each time step to approximate the intermediate bi-directional optical flows. These approximate flows, however, only work well in locally smooth regions and produce artifacts around motion boundaries. To address this shortcoming, we employ another U-Net to refine the approximated flow and also predict soft visibility maps. Finally, the two input images are warped and linearly fused to form each intermediate frame. By applying the visibility maps to the warped images before fusion, we exclude the contribution of occluded pixels to the interpolated intermediate frame to avoid artifacts. Since none of our learned network parameters are time-dependent, our approach is able to produce as many intermediate frames as needed. To train our network, we use 1,132 240-fps video clips, containing 300K individual video frames. Experimental results on several datasets, predicting different numbers of interpolated frames, demonstrate that our approach performs consistently better than existing methods.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/baa1ae74fbf7ed6204f2f6364d51375ff81aabc1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Video Enhancement with Task-Oriented Flow (2017, neutral): “…regression problems without representing motions, and have designed deep networks for frame interpolation (Mathieu et al 2016; Niklaus et al 2017a; Jiang et al 2017; Niklaus and Liu 2018), super-resolution (Huang et al 2015; Kappeler et al 2016; Tao et al 2017; Bulat et al 2018; Ahn et al 2018;…”
X3D: Expanding Architectures for Efficient Video Recognition (2020, methodology): “We follow previous works that use pre-computed proposals [24, 40, 66].”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, methodology): “[5], we assume the optical flow is locally smooth.”
VRT: A Video Restoration Transformer (2022, methodology): “[32], [70] and [3] are frame interpolation methods SuperSloMo, SepConv and DAIN, respectively.”
v2e: From Video Frames to Realistic DVS Events (2021, methodology): “C: Synthetic slow motion: The luma frames are then optionally interpolated using the Super-SloMo video interpolation network [14] to increase the temporal resolution of the input video.”
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement (2018, methodology): “Notably, the SuperSlomo method [38] generates the best results for the synthetic Urban sequence.”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “In addition, we train DVF [30] model and SuperSlomo [22] using our training pipeline on Vimeo90K dataset because the released models of these methods are trained on early datasets.”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “[22] and Bao et al .”
Multiscale Vision Transformers (2021, methodology): “Region-of-interest (RoI) features [36] are extracted at the generated feature map from MViT by extending a 2D proposal at a frame into a 3D RoI by replicating it along the temporal axis, similar as done in previous work [39, 89, 58], followed by application of frame-wise RoIAlign [43] and temporal g”
CoTracker: It is Better to Track Together (2023, background): “Here, we are interested in estimating point correspondences in videos containing dynamic objects and a moving camera.”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, neutral): “, video frame interpolation) [3, 22, 39, 62] is widely studied, respectively, spatial-temporal synthesis for dynamic scenes is relatively less explored [32, 79].”
Video Compression through Image Interpolation (2018, background): “Most image interpolation networks build on an encoder-decoder network architecture to move pixels through time [10, 11, 15, 19].”
Depth-Aware Video Frame Interpolation (2019, methodology): “We evaluate the PSNR and SSIM on the UCF101, Vimeo90K, and the HD datasets for comparisons.”
Self-supervised Video Representation Learning by Pace Prediction (2020, methodology): “It can be seen from the table that with the increase of the maximum pace, namely the number of training classes, the accuracy of the downstream action recognition task keeps increase, until p = [1, 4].”
Machine-learning-based spatio-temporal super resolution reconstruction of turbulent flows (2020, methodology): “2018), pixels motion transformation (Jiang et al. 2018), and neural network-based interpolation (Xu et al. 2020). For instance, Li et al. (2019) developed a machine learning based temporal super resolution technique called inbetweening to estimate the snapshot sequences between the start and last fr”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, background): “Video interpolation was the subject of a flurry of interest in the deep learning community a number of years ago [Niklaus et al., 2017, Jiang et al., 2018, Xue et al., 2019, Bao et al., 2019].”