Video Frame Synthesis Using Deep Voxel Flow

By Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala. 2017

809 citations recorded by the source corpus.

We address the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation). This problem is challenging because video appearance and motion can be highly complex. Traditional optical-flow-based solutions often fail where flow estimation is challenging, while newer neural-network-based methods that hallucinate pixel values directly often produce blurry results. We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow. Our method requires no human supervision, and any video can be used as training data by dropping, and then learning to predict, existing frames. The technique is efficient, and can be applied at any video resolution. We demonstrate that our method produces results that both quantitatively and qualitatively improve upon the state-of-the-art.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “Earlier work on video generation mostly focused on deterministic approaches (Srivastava et al., 2015; Carl Vondrick, 2016; Xingjian et al., 2015; Liu et al., 2017; Jia et al., 2016), which fail to capture the high degree of stochasticity inherent in video.”
Learning-based Video Motion Magnification (2018, methodology): “Recent techniques demonstrated high-quality results by explicitly shifting pixels using either optical flow [10,11,12] or pixelshifting convolution kernels [13,14].”
Stochastic Variational Video Prediction (2017, methodology): “Similar transformation-based models were also proposed by De Brabandere et al. (2016); Liu et al. (2017).”
Stochastic Adversarial Video Prediction (2018, background): “An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame [3,28,29,30,11,31,32,16,13,33,34,35].”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “The deep voxel ﬂow approach, a concurrent work to our paper, developed a deep neural network to output dense voxel ﬂows optimized frame interpolation re-sults [29].”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “This is of crucial importance in various contexts, such as weather forecasting [73], autonomous driving [29], reinforcement learning [43], robotics [16], or action recognition [33].”
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (2018, background): “The work of synthesizing lip motion from either audio [30, 28, 15, 16, 26, 11] or generating moving faces from videos [27, 24, 31, 21] has long been a task of concern in both the community of computer vision and graphics.”
StructureFlow: Image Inpainting via Structure-Aware Appearance Flow (2019, methodology): “Some papers [18, 23] manage to use multi-scale approaches to improve the results.”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “Early research uses either regression loss [19, 48] or GAN loss [37, 57, 78, 81] to supervise the frame prediction task.”
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement (2018, methodology): “We list the main difference with ﬂow-based methods [9], [36] and kernel-based approaches [12], [27] in Table 1.”
VideoFlow: A Flow-Based Generative Model for Video (2019, background): “…deterministic models focused on architectural changes, such as incorporating pixel transformations (Finn et al., 2016; De Brabandere et al., 2016; Liu et al., 2017) and predictive 1We generate 64x64 videos of 20 frames in less than 3.5 seconds on a NVIDIA P100 GPU. coding architectures (Lotter…”
Self-Supervised Learning via Conditional Motion Propagation (2019, background): “Other works [18, 16, 21, 33] learn representations by synthesizing frames or predicting correct temporal order.”
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation (2017, methodology): “[15] develop a CNN model for frame interpolation that has an explicit sub-network for motion estimation.”
Detail-Revealing Deep Video Super-Resolution (2017, methodology): “Since we do not have ground truth flow, unsupervised warping loss is used as [21, 30]”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, background): “[42] propose to learn a 3D optical flow in”