High-Fidelity Generative Image Compression

By Fabian Mentzer, George Toderici, Michael Tschannen, Eirikur Agustsson. 2020

640 citations recorded by the source corpus.

We extensively study how to combine Generative Adversarial Networks and learned compression to obtain a state-of-the-art generative lossy compression system. In particular, we investigate normalization layers, generator and discriminator architectures, training strategies, as well as perceptual losses. In contrast to previous work, i) we obtain visually pleasing reconstructions that are perceptually similar to the input, ii) we operate in a broad range of bitrates, and iii) our approach can be applied to high-resolution images. We bridge the gap between rate-distortion-perception theory and practice by evaluating our approach both quantitatively with various perceptual metrics and a user study. The study shows that our method is preferred to previous approaches even if they use more than 2x the bitrate.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “In particular, [49] and [21] have shown that adding an adversarial realism prior to the usual autoencoder objective helps to produce more realistic images at higher compression rates by locally trading reconstruction fidelity for realism.”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Neural Compression [9], [297], [435] instead expliots generative models to optimize the storage overhead of input data without much distortion, which can benefit both efficacy”
SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “This is aligned with recent ﬁndings in the ﬁeld of neural image compression [61], which also adopt a lighter encoder and a heavier decoder.”
Memory Replay with Data Compression for Continual Learning (2022, positive): “On the other hand, neural compression approaches generally rely on optimizing Shannon’s rate-distortion trade-off, through RNNs (Toderici et al., 2015; 2017), auto-encoders (Agustsson et al., 2017) and GANs (Mentzer et al., 2020).”
CompressAI: a PyTorch library and evaluation platform for end-to-end compression research (2020, neutral): “Following the success of deep neural networks in out-performing most conventional approaches in computer vision applications, Artiﬁcial Neural Network (ANN) based codecs have recently demonstrated impressive results for compressing images [1–5].”
Nonlinear Transform Source-Channel Coding for Semantic Communications (2021, methodology): “Compared to the PSNR results, we can ﬁnd that the BPG series are inferior to the learning-based schemes because BPG compression is designed to be optimized for squared error with hand-selected constraints [21].”
The Devil Is in the Details: Window-based Attention for Image Compression (2022, methodology): “We have conducted some comparative experiments by replacing GDN with batch normalization [22], layer normalization [3], and channel normalization [33].”
Lossy Compression for Lossless Prediction (2021, background): “Most research in neural compression is either focused on estimation and optimization of the rate term [81, 105–111] or on developing perceptually meaningful distortions [8, 10, 112, 113].”
Finite Scalar Quantization: VQ-VAE Made Simple (2023, background): “Neural compression Many works (Ball´e et al., 2016; Minnen et al., 2018; Lu et al., 2019; Mentzer et al., 2020; Cheng et al., 2020) rely on unbounded scalar quantization and constrain the entropy of the quantized representation to prevent spreading to all integers.”
A Streamwise Gan Vocoder for Wideband Speech Coding at Very Low Bit Rate (2021, positive): “We replace instance normalization with channel normalization [18], that estimates the statistics along the channel dimension instead.”
ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding (2022, neutral): “We will also delve into models with both objective and subjective image quality remarkable, following exiting literature [10,18,38,43].”
Deblurring via Stochastic Refinement (2021, methodology): “Because our datasets do not have enough examples to reliably compute FID and KID, we extract 15 non-overlapping patches of size 256 × 240 from each 1280 × 720 image and compute the Inception-based metrics at the patch level, similar to [48].”
Toward Semantic Communications: Deep Learning-Based Image Semantic Coding (2022, methodology): “loss of GANs [32], [33] captures the distribution of natural images and coincides with human perception.”