Learned Video Compression

By Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G. Anderson, Lubomir D. Bourdev. 2018

261 citations recorded by the source corpus.

We present a new algorithm for video coding, learned end-to-end for the low-latency mode. In this setting, our approach outperforms all existing video codecs across nearly the entire bitrate range. To our knowledge, this is the first ML-based method to do so. We evaluate our approach on standard video compression test sets of varying resolutions, and benchmark against all mainstream commercial codecs in the low-latency mode. On standard-definition videos, HEVC/H.265, AVC/H.264 and VP9 typically produce codes up to 60% larger than our algorithm. On high-definition 1080p videos, H.265 and VP9 typically produce codes up to 20% larger, and H.264 up to 35% larger. Furthermore, our approach does not suffer from blocking artifacts and pixelation, and thus produces videos that are more visually pleasing. We propose two main contributions. The first is a novel architecture for video compression, which (1) generalizes motion estimation to perform any learned compensation beyond simple translations, (2) rather than strictly relying on previously transmitted reference frames, maintains a state of arbitrary information learned by the model, and (3) enables jointly compressing all transmitted signals (such as optical flow and residual). Secondly, we present a framework for ML-based spatial rate control --- a mechanism for assigning variable bitrates across space for each frame. This is a critical component for video coding, which to our knowledge had not been developed within a machine learning setting.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Signal Processing

Citations in the corpus

High Fidelity Neural Audio Compression (2022, positive): “Leveraging neural networks as trained transforms via an encoder-decoder mechanism has been explored by Morishima et al. (1990); Rippel et al. (2019); Zeghidour et al. (2021).”
Scale-Space Flow for End-to-End Optimized Video Compression (2020, methodology): “264 in terms of PSNR and that outperforms HEVC in terms of MS-SSIM [24, 30].”
Deep Contextual Video Compression (2021, methodology): “For video compression, recent works in [15], [16], and [12, 17] have some investigations on condition coding.”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, background): “Recently, learning based video compression [38, 22, 26, 12, 10, 19, 21, 23, 14, 11, 2, 42, 8, 39, 20, 17, 41] has become a new research direction.”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “DNN aided compression algorithms have achieved significant results, often outperforming state-of-the-art standardised codecs in a variety of source domains, from image [155], [156], [157], [158], [159], to video [160], [161], [162], [163], [164], [165], [166], [167], [168], speech [169], [170], and ”
Video Compression With Rate-Distortion Autoencoders (2019, background): “Learned Video Compression Video compression shares many similarities with image compression, but the large size of video data, and the very high degree of redundancy create new challenges [15, 30, 33, 40].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “A number of recent works [3, 16, 26, 43, 52, 65, 87] propose using a deep network to compress arbitrary videos.”