Softmax Splatting for Video Frame Interpolation

By Simon Niklaus, Feng Liu. Computer Vision and Pattern Recognition, 2020

473 citations recorded by the source corpus.

Differentiable image sampling in the form of backward warping has seen broad adoption in tasks like depth estimation and optical flow prediction. In contrast, how to perform forward warping has seen less attention, partly due to additional challenges such as resolving the conflict of mapping multiple pixels to the same target location in a differentiable way. We propose softmax splatting to address this paradigm shift and show its effectiveness on the application of frame interpolation. Specifically, given two input frames, we forward-warp the frames and their feature pyramid representations based on an optical flow estimate using softmax splatting. In doing so, the softmax splatting seamlessly handles cases where multiple source pixels map to the same target location. We then use a synthesis network to predict the interpolation result from the warped representations. Our softmax splatting allows us to not only interpolate frames at an arbitrary time but also to fine tune the feature pyramid and the optical flow. We show that our synthesis approach, empowered by softmax splatting, achieves new state-of-the-art results for video frame interpolation.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/82da38e2ddd8aebbc13d9e4505bc86ad83c0d6da

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Most approaches for interpolating between video frames work in 2D image space, by directly predicting kernels that blend two images [51, 52, 53], or by modeling optical flow and warping frames/features [2, 31, 49, 50].”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, methodology): “We provide a pair of experiments to show L Lap [40,41] is quantitatively better than L 1 .”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “Video synthesis involves a wide range of tasks including video-to-video translation [73], image animation [64, 65], frame interpolation [52, 4, 53], and video prediction.”
BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, unknown): “Grid-like designs are seen in various vision tasks such as object detection [26, 30], semantic segmentation [7, 26, 30, 40], and frame interpolation [23].”
Deep Contextual Video Compression (2021, background): “Other video tasks show that utilizing temporal information as condition is helpful [27, 28].”