Generative Adversarial Networks for Extreme Learned Image Compression

By Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, Luc Van Gool. 2018

667 citations recorded by the source corpus.

We present a learned image compression system based on GANs, operating at extremely low bitrates. Our proposed framework combines an encoder, decoder/generator and a multi-scale discriminator, which we train jointly for a generative learned compression objective. The model synthesizes details it cannot afford to store, obtaining visually pleasing results at bitrates where previous methods fail and show strong artifacts. Furthermore, if a semantic label map of the original image is available, our method can fully synthesize unimportant regions in the decoded image such as streets and trees from the label map, proportionally reducing the storage cost. A user study confirms that for low bitrates, our approach is preferred to state-of-the-art methods, even when they use more than double the bits.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, neutral): “Using GANs for extremely low rate compression is explored in [35] and [1].”
High-Fidelity Generative Image Compression (2020, methodology): “Furthermore, adversarial losses were previously employed to improve the visual quality of neural compression systems [39, 40, 48, 3, 9].”
The Perception-Distortion Tradeoff (2017, background): “GAN-based image restoration Most recently, GAN-based methods have demonstrated unprecedented perceptual quality in super-resolution [3], [6], [9], [37], inpainting [7], [20], [38], compression [21], [39], [40], deblurring [41] and image-to-image translation [11], [22], [42].”
Learned Video Compression (2018, background): “In the last two years, we have seen a great surge of ML-based image compression approaches [13, 39, 40, 5, 4, 12, 20, 38, 33, 18, 2, 22, 6, 3, 8, 27, 28].”
The GAN Landscape: Losses, Architectures, Regularization, and Normalization (2018, background): “• CIN: Use c i for the discriminator ResBlock hidden layer output channels.”
Channel-Wise Autoregressive Entropy Models for Learned Image Compression (2020, background): “Interesting future research could look at combining our entropy model with more sophisticated image quality metrics such as perceptual metrics [33]– [35] and adversarial loss [36]–[38].”
Deep Generative Models for Distribution-Preserving Lossy Compression (2018, methodology): “Work [4] leveraged GANs for artifact suppression, whereas [15] used them to learn synthesizing image content beyond local texture, such as facades of buildings, obtaining visually pleasing results at very low bitrates.”
Finite Scalar Quantization: VQ-VAE Made Simple (2023, methodology): “Bounded scalar quantization ( i.e. , FSQ), has been used to represent images with high ﬁdelity (Mentzer et al. (2018) use d =16 , L =5 ), and for “extreme compression” (Agustsson et al. (2019) used d =5 , L =5 ).”
Semantic Communications: Principles and Challenges (2021, background): “[62] introduced the GAN based extreme learned image compression, which obtains visually pleasing results at an extremely low bit rate.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “We note that there are various other variants of LFQ, e.g ., opting for the multivariant over the binary codebook C i or employing other quantization techniques such as Agustsson et al. (2019).”
Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules (2020, background): “Some approaches [12, 13, 14] use generative models to learn the distribution of images using adversarial training to achieve better subjective quality at extremely low bit rate.”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, methodology): “In our work, instead of following the existing works [22, 14, 3, 19] to adopt the pixel-level operations as in the traditional video codecs, we propose a new feature-space video coding network (referred to as FVC) by reducing the spatial-temporal redundancy in the feature space.”
Context-adaptive Entropy Model for End-to-end Optimized Image Compression (2018, background): “First, as a consequence of the recent success of generative models, some image compression approaches targeting the superior perceptual quality (Agustsson et al., 2018; Santurkar et al., 2018; Rippel & Bourdev, 2017) have been proposed.”
Checkerboard Context Model for Efficient Learned Image Compression (2021, background): “Many studies aim to establish novel image compression pipelines based on recurrent neural networks [35], convolutional autoencoders [5, 34, 31, 3, 6], or generative adversarial networks [1].”
DVC: An End-To-End Deep Video Compression Framework (2018, methodology): “Recently, deep neural network (DNN) based autoencoder for image compression [34, 11, 35, 8, 12, 19, 33, 21, 28, 9] has achieved comparable or even better performance than the traditional image codecs like JPEG [37], JPEG2000 [29] or BPG [1].”
Lossy Compression for Lossless Prediction (2021, background): “Most research in neural compression is either focused on estimation and optimization of the rate term [93, 100–106] or on developing perceptually meaningful distortions [8, 10, 107, 108].”
Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks (2019, methodology): “Generative Adversarial Networks GANs [9] have successfully been applied to many computer vision tasks, such as image super-resolution [21] and image compression [1], and a myriad of further variants have been proposed in recent years (e.”
Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff (2019, methodology): “For example, incorporating generative adversarial type losses has been shown to lead to significantly better perceptual quality, but at the cost of increased distortion (Tschannen et al., 2018; Agustsson et al., 2018; Santurkar et al., 2018).”
Practical Full Resolution Learned Lossless Image Compression (2018, background): “While it is known that likelihood-based discrete generative models can in theory be used for lossless compression [37], recent work on learned compression using deep neural networks has solely focused on lossy compression [4, 38, 39, 32, 1, 3].”