End-to-end optimization of nonlinear transform codes for perceptual quality

By Johannes Ballé, Valero Laparra, Eero P. Simoncelli. 2016

273 citations recorded by the source corpus.

We introduce a general framework for end-to-end optimization of the rate-distortion performance of nonlinear transform codes assuming scalar quantization. The framework can be used to optimize any differentiable pair of analysis and synthesis transforms in combination with any differentiable perceptual metric. As an example, we consider a code built from a linear transform followed by a form of multi-dimensional local gain control. Distortion is measured with a state-of-the-art perceptual metric. When optimized over a large database of images, this representation offers substantial improvements in bitrate and perceptual appearance over fixed (DCT) codes, and over linear transform codes optimized for mean squared error.

Subfields: Computer Vision and Pattern Recognition, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Optimization & Training

In Signal Processing

Citations in the corpus

End-to-end Optimized Image Compression (2016, methodology): “This scheme is called transform coding due to the central role of the transforma-∗ (Ball´e, Laparra, and Simoncelli, 2016).”
Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations (2017, methodology): “Our training set is composed similarly to that described in [4].”
Full Resolution Image Compression with Recurrent Neural Networks (2016, methodology): “Most image compression neural networks use a ﬁxed compression rate based on the size of a bottleneck layer [2].”
Lossy Image Compression with Compressive Autoencoders (2017, methodology): “The transformations used by Ball ´ e et al. (2016) consist of a single linear layer combined with a form of contrast gain control, while our framework relies on more standard deep convolutional neural networks.”
Generative Adversarial Networks for Extreme Learned Image Compression (2018, methodology): “To reduce coding rates, many deep compression systems rely on context models to capture the distribution of the bit stream (Ball´e et al., 2016b; Toderici et al., 2016; Li et al., 2017; Rippel & Bourdev, 2017; Mentzer et al., 2018).”
Image and Video Compression With Neural Networks: A Review (2019, methodology): “first introduced an end-to-end optimized CNN framework for image compression under the scalar quantization assumption in 2016 [52], [53].”
Online Learned Continual Compression with Adaptive Quantization Modules (2019, methodology): “Works by Theis et al. (2017); Ball´e et al. (2016); Johnston et al. (2018) have shown learned compressions can outperform standard algorithms like JPEG.”
NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing (2018, unknown): “…which compresses an input document into compact binary codes, can be casted as a conventional ratedistortion tradeoff problem (Theis et al., 2017; Ballé et al., 2016): min − log2R(z)︸︷︷︸ Rate +β ·D(x, x̂)︸︷︷︸ Distortion , (7) where rate and distortion denote the effective code length, i.e.,…”
Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff (2019, background): “…is seeing a surge of research, due in part to the advancements in deep learning and their application in this domain (Toderici et al., 2016; 2017; Ball´e et al., 2016; 2017; 2018; Agustsson et al., 2017; 2018; Rippel & Bour-dev, 2017; Minnen et al., 2018; Li et al., 2018; Mentzer et al., 2018;…”
Variational image compression with a scale hyperprior (2018, methodology): “Note that Ballé et al. (2016b) compare models trained for different metrics, but their results are limited by the choice of transforms.”
Soft-to-Hard Vector Quantization for End-to-End Learned Compression of Images and Neural Networks (2017, methodology): “The works of (Ballé et al., 2016b) build a deep model using multiple generalized divisive normalization (GDN) layers and their inverses (IGDN), which are specialized layers designed to capture local joint statistics of natural images.”
Conditional Probability Models for Deep Image Compression (2018, background): “Other notable techniques involve progressive encoding/decoding strategies [19, 20], adversarial training [14], multi-scale image decompositions [14], and generalized divisive normalization (GDN) layers [4, 3].”