Variational image compression with a scale hyperprior

By Johannes Ballé, David C. Minnen, Saurabh Singh, Sung Jin Hwang, Nick Johnston. 2018

2,395 citations recorded by the source corpus.

We describe an end-to-end trainable model for image compression based on variational autoencoders. The model incorporates a hyperprior to effectively capture spatial dependencies in the latent representation. This hyperprior relates to side information, a concept universal to virtually all modern image codecs, but largely unexplored in image compression using artificial neural networks (ANNs). Unlike existing autoencoder compression methods, our model trains a complex prior jointly with the underlying autoencoder. We demonstrate that this model leads to state-of-the-art image compression when measuring visual quality using the popular MS-SSIM index, and yields rate-distortion performance surpassing published ANN-based methods when evaluated using a more traditional metric based on squared error (PSNR). Furthermore, we provide a qualitative comparison of models trained for different distortion metrics.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, background): “Other researchers have developed practical lossy coding schemes achieving variational rate distortion bounds for particular latent variable models which exploited the factorial structure of the variational posterior (Ballé et al., 2018; Theis et al., 2017; Yang et al., 2020).”
High-Fidelity Generative Image Compression (2020, methodology): “density of the auto-encoder latents for more efficient arithmetic coding, using hierarchical priors, auto-regression with various context shapes, or a combination thereof [6, 31, 28, 39, 32, 26, 33].”
Learned Video Compression (2018, neutral): “proaches [15, 44, 45, 5, 4, 14, 25, 43, 38, 23, 2, 27, 6, 3, 10, 32, 33].”
Generative Adversarial Networks for Extreme Learned Image Compression (2018, methodology): “The so-obtained model has a bitrate of 0.07 bpp and gets a slightly better MS-SSIM than BPG at the same bpp on the validation set.”
Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff (2019, background): “Over the past decades, there has been a constant search for more “perceptual” distortion criteria, ranging from the simple SSIM/MS-SSIM (Wang et al., 2003; 2004) to the recently popular deep-feature based distortion (Johnson et al., 2016; Zhang et al., 2018).”
Variable Rate Deep Image Compression With a Conditional Autoencoder (2019, methodology): “[4] (optimized for MS-SSIM) Rippel & Bourdev [19] Mentzer et al.”
Improving Inference for Neural Image Compression (2020, methodology): “This step is simplified by a clever restriction on the shape of p, such that it agrees with the discretized P on all integers (see [Ballé et al., 2017, 2018, Minnen et al., 2018]).”
Lossy Compression for Lossless Prediction (2021, background): “’s [28] hyperprior entropy model with a learned rounded precision.”
Practical Full Resolution Learned Lossless Image Compression (2018, background): “Further, following [5] we discard high saturation/ non-photographic images, i.”
ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding (2022, positive): “The training settings are accordingly sketched from existing literature [5,15,24,39].”
COIN: COmpression with Implicit Neural representations (2021, methodology): “We compare our model against three autoencoder based neural compression baselines which we refer to as BMS [2], MBT [24], and CST [7].”
End-to-End Learnt Image Compression via Non-Local Attention Optimization and Improved Context Modeling (2019, methodology): “Our NLAIC is built on a variational autoencoder structure [10], with non-local attention modules”
Scale-Space Flow for End-to-End Optimized Video Compression (2020, methodology): “Image Compression Research on learning-based image compression [7, 10, 32, 4, 29, 8, 25, 5] has shown significant progress in terms of rate–distortion performance compared to standard codecs such as JPEG [34], JPEG2000 [21] and BPG [11].”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, methodology): “[97] propose a supervised compression method for resource-constrained edge computing systems, which adapts ideas from knowledge distillation and neural image compression [4, 5].”
Neural Video Compression with Diverse Contexts (2023, methodology): “Most neural image codecs are based on hyperprior [4] where some bits are first used to provide basic contexts for entropy coding.”
V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction (2020, methodology): “We adapt Ball´e et al. ’s variational image compression algorithm [2] to compress our intermediate representations; a convolutional network learns to compress our representations with the help of a learned hyperprior.”
Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules (2020, methodology): “2) Minnen [MSE] (NIPS18) [20] Lee [MSE] (ICLR19) [21] Ballé [MSE] (ICLR18) [19] Ballé [MS-SSIM] (ICLR18) [19] Li (CVPR2018) [15] HEVC-intra (BPG) JPEG2000 (OpenJPEG) JPEG”
Deep Generative Models for Distribution-Preserving Lossy Compression (2018, background): “Compressive autoencoders [3, 5, 6, 8, 13, 10] and recurrent neural networks (RNNs) [1, 2, 7] have emerged as the most popular DNN architectures for compression.”
Practical Lossless Compression with Latent Variables using Bits Back Coding (2019, methodology): “, 1987) and the more recently developed asymmetric numeral systems (Duda, 2009) are actual algorithms for implementing lossless compression, given some kind of probabilistic model. The field of machine learning has experienced an explosion of activity in recent years, and we have seen a number of pa”