Video Compression With Rate-Distortion Autoencoders

By AmirHossein Habibian, Ties van Rozendaal, J. Tomczak, Taco Cohen. 2019

238 citations recorded by the source corpus.

In this paper we present a a deep generative model for lossy video compression. We employ a model that consists of a 3D autoencoder with a discrete latent space and an autoregressive prior used for entropy coding. Both autoencoder and prior are trained jointly to minimize a rate-distortion loss, which is closely related to the ELBO used in variational autoencoders. Despite its simplicity, we find that our method outperforms the state-of-the-art learned video compression networks based on motion compensation or interpolation. We systematically evaluate various design choices, such as the use of frame-based or spatio-temporal autoencoders, and the type of autoregressive prior. In addition, we present three extensions of the basic method that demonstrate the benefits over classical approaches to compression. First, we introduce \emph{semantic compression}, where the model is trained to allocate more bits to objects of interest. Second, we study \emph{adaptive compression}, where the model is adapted to a domain with limited variability, \eg videos taken from an autonomous car, to achieve superior compression on that domain. Finally, we introduce \emph{multimodal compression}, where we demonstrate the effectiveness of our model in joint compression of multiple modalities captured by non-standard imaging sensors, such as quad cameras. We believe that this opens up novel video compression applications, which have not been feasible with classical codecs.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Eirikur Agustsson
2. Luc Van Gool
3. Bin Li
4. Yan Lu
5. Jiahao Li
6. Dong Xu
7. David C. Minnen
8. George Toderici

In Generative Models

1. Eirikur Agustsson
2. David C. Minnen
3. Jiahao Li
4. Yan Lu
5. Bin Li
6. George Toderici
7. Guo Lu
8. Dong Xu

Citations in the corpus

• Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “Following (van den Oord et al., 2017; Habibian et al., 2019; Razavi et al., 2019) we model the latent distribution P( z ) using an ARM.”
• Diffusion Priors In Variational Autoencoders (2021, background): “For example, Habibian et al. (2019) and Chen et al. (2017) respectively showed that autoregressive models and normalizing flows (Rezende & Mohamed, 2015, NFs) greatly improve the performance of VAEs when used as prior distributions.”
• FVC: A New Framework towards Deep Video Compression in Feature Space (2021, background): “Recently, learning based video compression [38, 22, 26, 12, 10, 19, 21, 23, 14, 11, 2, 42, 8, 39, 20, 17, 41] has become a new research direction.”
• Dynamical Variational Autoencoders: A Comprehensive Review (2020, methodology): “Other examples include VQ-VAE (that is a mix of VAE and vector quantization of z) applied to speech coding (van den Oord et al., 2017; Gârbacea et al., 2019), and video compression with rate-distortion autoencoders (Habibian et al., 2019).”
• Improving Inference for Neural Image Compression (2020, background): “It is so far believed that bits-back coding is incompatible with lossy compression [Habibian et al., 2019] because it requires inference on the decoder side, which does not have access to the exact (undistorted) input data.”
• Scale-Space Flow for End-to-End Optimized Video Compression (2020, methodology): “Specifically, for equal PSNR, our method provides an average Bjøntegaard Delta (BD) rate reduction [11] of 13.4% compared to [21] and a savings of 42.9% over [35], while we see a 30.3% savings over [16] for equal MS-SSIM (see Section 5 for a detailed evaluation).”
• Deep Contextual Video Compression (2021, background): “In addition, 3D autoencoder was proposed to encode group of pictures in [24, 25].”
• CompressAI: a PyTorch library and evaluation platform for end-to-end compression research (2020, background): “Similarly, significant progresses have been reported regarding neural networks targeting video compression [17–19].”
• Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “DNN aided compression algorithms have achieved significant results, often outperforming state-of-the-art standardised codecs in a variety of source domains, from image [155], [156], [157], [158], [159], to video [160], [161], [162], [163], [164], [165], [166], [167], [168], speech [169], [170], and ”