Video Compression through Image Interpolation

By Chao-Yuan Wu, Nayan Singhal, Philipp Krahenbuhl. 2018

352 citations recorded by the source corpus.

An ever increasing amount of our digital communication, media consumption, and content creation revolves around videos. We share, watch, and archive many aspects of our lives through them, all of which are powered by strong video compression. Traditional video compression is laboriously hand designed and hand optimized. This paper presents an alternative in an end-to-end deep learning codec. Our codec builds on one simple idea: Video compression is repeated image interpolation. It thus benefits from recent advances in deep image interpolation and generation. Our deep video codec outperforms today's prevailing codecs, such as H.261, MPEG-4 Part 2, and performs on par with H.264.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learned Video Compression (2018, background): “[52] first encodes key frames, and proceeds to hierarchically interpolate the frames between them.”
Deep Generative Models for Distribution-Preserving Lossy Compression (2018, methodology): “These methods were shown to outperform traditional compression standards in image compression [1–10], speech compression [11], and video compression [12] under several distortion measures.”
Video Compression With Rate-Distortion Autoencoders (2019, methodology): “The method described in [40] compresses videos by first encoding key frames, and then interpolating them in a hi-”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “A number of recent works [3, 16, 26, 43, 52, 65, 87] propose using a deep network to compress arbitrary videos.”
DVC: An End-To-End Deep Video Compression Framework (2018, methodology): “In addition, a learning based video compression system in [40], denoted by Wu ECCV2018, is also included for comparison.”
Deep Contextual Video Compression (2021, background): “For example, Wu et al. [10] proposed interpolating the predicted frame from previous and future frames, and then the frame residue is encoded.”
Neural Video Compression with Diverse Contexts (2023, background): “Mainstream frameworks of NVC can be classified into residual coding-based [1,13,31,32,34,36,47,59,61] and condition coding-based [21, 27–29, 33, 38, 50].”
Softmax Splatting for Video Frame Interpolation (2020, background): “Frame interpolation can also support inter-frame compression for videos [49], serve as an auxiliary task for optical ﬂow estimation [30, 50], or generate training data to learn how to synthesize motion blur [6].”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, background): “Recently, learning based video compression [38, 22, 26, 12, 10, 19, 21, 23, 14, 11, 2, 42, 8, 39, 20, 17, 41] has become a new research direction.”
Scale-Space Flow for End-to-End Optimized Video Compression (2020, methodology): “To ensure we perform an apples-to-apples comparison with recent methods, we also evaluate HEVC with the settings used in [24, 19, 38] to verify the baseline matches what is reported in those papers, which we refer to as HEVC (very-fast).”
Deep Probabilistic Video Compression (2018, methodology): “Wu et al. (2018) perform video compression through image interpolation between reference frames using a predictive model based on a deep neural network.”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, neutral): “DNN aided compression algorithms have achieved significant results, often outperforming state-of-the-art standardised codecs in a variety of source domains, from image [155], [156], [157], [158], [159], to video [160], [161], [162], [163], [164], [165], [166], [167], [168], speech [169], [170], and ”
NeRV: Neural Representations for Videos (2021, background): “Similarly, [24] converted the video compression problem into an image interpolation problem and proposed an interpolation network, resulting in competitive compression quality.”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, background): “VFI supports various applications like slow-motion generation, video compression [57], and video frame predition [58].”