Joint Autoregressive and Hierarchical Priors for Learned Image Compression

By David C. Minnen, Johannes Ballé, George Toderici. 2018

1,660 citations recorded by the source corpus.

Recent models for learned image compression are based on autoencoders that learn approximately invertible mappings from pixels to a quantized latent representation. The transforms are combined with an entropy model, which is a prior on the latent representation that can be used with standard arithmetic coding algorithms to generate a compressed bitstream. Recently, hierarchical entropy models were introduced as a way to exploit more structure in the latents than previous fully factorized priors, improving compression performance while maintaining end-to-end optimization. Inspired by the success of autoregressive priors in probabilistic generative models, we examine autoregressive, hierarchical, and combined priors as alternatives, weighing their costs and benefits in the context of image compression. While it is well known that autoregressive models can incur a significant computational penalty, we find that in terms of compression performance, autoregressive and hierarchical priors are complementary and can be combined to exploit the probabilistic structure in the latents better than all previous learned models. The combined model yields state-of-the-art rate–distortion performance and generates smaller files than existing methods: 15.8% rate reductions over the baseline hierarchical model and 59.8%, 35%, and 8.4% savings over JPEG, JPEG2000, and BPG, respectively. To the best of our knowledge, our model is the first learning-based method to outperform the top standard image codec (BPG) on both the PSNR and MS-SSIM distortion metrics.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Efficiency & Compression

In Signal Processing

Citations in the corpus

Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “For lossy image compression, [24] trains hierarchical and autoregressive priors jointly to improve the entropy coding part of the compression system.”
High-Fidelity Generative Image Compression (2020, methodology): “density of the auto-encoder latents for more efficient arithmetic coding, using hierarchical priors, auto-regression with various context shapes, or a combination thereof [6, 31, 28, 39, 32, 26, 33].”
Learned Video Compression (2018, background): “In the last two years, we have seen a great surge of ML-based image compression approaches [13, 39, 40, 5, 4, 12, 20, 38, 33, 18, 2, 22, 6, 3, 8, 27, 28].”
Generative Adversarial Networks for Extreme Learned Image Compression (2018, methodology): “Given the dramatic bitrate savings we achieve according to the user study (BPG needing 21–181% more bits), and the competitiveness of BPG to the most recent state-of-theart [31], we conclude that our proposed system presents a 0.”
Lossy Compression for Lossless Prediction (2021, methodology): “Later approaches improved upon the original model by adding a hierarchical layers and autoregressive components (Ballé et al., 2018; Minnen et al., 2018; Lee et al., 2019; Johnston et al., 2018).”
Channel-Wise Autoregressive Entropy Models for Learned Image Compression (2020, methodology): “7% over the context-adaptive baseline [13].”
Nonlinear Transform Source-Channel Coding for Semantic Communications (2021, neutral): “To seek computational parallelism, in this paper, we focus on the FA mode, a better performance backward adaptation (BA) mode with higher processing latency will be discussed in the future [27].”
Improving Inference for Neural Image Compression (2020, methodology): “We tested two variants of our method ([M1] and [M2]): SGA builds on the exact same model and training procedure as in [Minnen et al., 2018] (the Base Hyperprior model [M3]) and changes only how the trained model is used for compression by introducing hybrid amortized-iterative inference [Campos et a”
Improved Autoregressive Modeling with Distribution Smoothing (2021, background): “For instance, they have shown success in compressing images (Minnen et al., 2018), synthesizing speech (Oord et al., 2016a) and modeling complex decision rules in games (Vinyals et al., 2019).”
Deep Probabilistic Video Compression (2018, background): “Interestingly, in the context of image compression, deep learning approaches that are fundamentally different to existing codecs have already shown promising results (Ballé et al., 2018; 2016; Theis et al., 2017; Agustsson et al., 2017; Minnen et al., 2018), mostly within the past year.”
Practical Lossless Compression with Latent Variables using Bits Back Coding (2019, methodology): “Ball´e et al. (2017), Theis et al. (2017), Ball´e et al. (2018), and Minnen et al. (2018) all implement lossy compression using (variational) auto-encoder style models, and Tschannen et al. (2018) train a model for lossy compression using a GAN-like objec-tive.”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, methodology): “Other methods [4, 5, 25, 29] adopted the auto-encoder structure to first encode images as latent representations, and then decode the latent representations from the feature space to the pixel space, which have achieved the state-of-the-art performance for image compression.”
Variable Rate Deep Image Compression With a Conditional Autoencoder (2019, background): “In Figure 9, we compare our variable-rate networks optimized for MSE, MS-SSIM and combined MSE+MS-SSIM losses, respectively.”
Scale-Space Flow for End-to-End Optimized Video Compression (2020, methodology): “For each of the three latent types, z0,vi,wi we use a separate hyperprior [8, 26] to model the corresponding density.”
Neural Video Compression with Diverse Contexts (2023, methodology): “If considering the 8 spatial neighbours for a position, the auto-regressive model [40] uses 4 (left, top-left, top, top-right) neighbours for every position if not considering the boundary region.”
Finite Scalar Quantization: VQ-VAE Made Simple (2023, background): “Neural compression Many works (Ball´e et al., 2016; Minnen et al., 2018; Lu et al., 2019; Mentzer et al., 2020; Cheng et al., 2020) rely on unbounded scalar quantization and constrain the entropy of the quantized representation to prevent spreading to all integers.”