Scale-Space Flow for End-to-End Optimized Video Compression

By Eirikur Agustsson, David C. Minnen, Nick Johnston, Johannes Ballé, Sung Jin Hwang, George Toderici. Computer Vision and Pattern Recognition, 2020

384 citations recorded by the source corpus.

Despite considerable progress on end-to-end optimized deep networks for image compression, video coding remains a challenging task. Recently proposed methods for learned video compression use optical flow and bilinear warping for motion compensation and show competitive rate-distortion performance relative to hand-engineered codecs like H.264 and HEVC. However, these learning-based methods rely on complex architectures and training schemes including the use of pre-trained optical flow networks, sequential training of sub-networks, adaptive rate control, and buffering intermediate reconstructions to disk during training. In this paper, we show that a generalized warping operator that better handles common failure cases, e.g. disocclusions and fast motion, can provide competitive compression results with a greatly simplified model and training procedure. Specifically, we propose scale-space flow, an intuitive generalization of optical flow that adds a scale parameter to allow the network to better model uncertainty. Our experiments show that a low-latency video compression model (no B-frames) using scale-space flow for motion compensation can outperform analogous state-of-the art learned video compression models while being trained using a much simpler procedure and without any pre-trained optical flow networks.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/29efad9e507843ea3481cbddce9f49fad6684e41

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

NeRV: Neural Representations for Videos (2021, background): “We compare with H.264 [8], HEVC [2], STAT-SSF-SP [56], HLVC [26], Scale-space [25], and Wu et al .”
CompressAI: a PyTorch library and evaluation platform for end-to-end compression research (2020, background): “Similarly, significant progresses have been reported regarding neural networks targeting video compression [17–19].”
Diffusion Probabilistic Modeling for Video Generation (2022, methodology): “Similar observations have been made in neural video compression [9,26].”
Deep Contextual Video Compression (2021, neutral): “According to the performance comparison in [37, 38, 31], DVCPro[4] is one previous SOTA DLbased codec among recent works [8, 6, 9, 5, 26].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “A number of recent works [3, 16, 26, 43, 52, 65, 87] propose using a deep network to compress arbitrary videos.”