Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff

By Yochai Blau, Tomer Michaeli. 2019

444 citations recorded by the source corpus.

Lossy compression algorithms are typically designed and analyzed through the lens of Shannon's rate-distortion theory, where the goal is to achieve the lowest possible distortion (e.g., low MSE or high SSIM) at any given bit rate. However, in recent years, it has become increasingly accepted that "low distortion" is not a synonym for "high perceptual quality", and in fact optimization of one often comes at the expense of the other. In light of this understanding, it is natural to seek for a generalization of rate-distortion theory which takes perceptual quality into account. In this paper, we adopt the mathematical definition of perceptual quality recently proposed by Blau & Michaeli (2018), and use it to study the three-way tradeoff between rate, distortion, and perception. We show that restricting the perceptual quality to be high, generally leads to an elevation of the rate-distortion curve, thus necessitating a sacrifice in either rate or distortion. We prove several fundamental properties of this triple-tradeoff, calculate it in closed form for a Bernoulli source, and illustrate it visually on a toy MNIST example.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Efficiency & Compression

Citations in the corpus

On Mutual Information Maximization for Representation Learning (2019, background): “…et al. (2018) similarly observed that models achieving the same evidence lower bound value can lead to vastly different representations depending on the employed encoder architecture, and do not necessarily capture useful information about the data (Tschannen et al., 2018; Blau and Michaeli, 2019).”
High-Fidelity Generative Image Compression (2020, methodology): “While [40] provides a proofof-concept implementation of a low-resolution compression system with a GAN discriminator as decoder, [48, 9] focus on incorporating a GAN loss into the rate-distortion objective in a conceptually sound fashion.”
The Perception-Distortion Tradeoff (2017, background): “Theoretically, its effect can be studied through the rate-distortion-perception function [55], [56], [57], which is an extension of the ratedistortion function (16) and the perception-distortion function (11), characterizing the triple tradeoff between rate, distortion, and perceptual quality.”
Self-Supervised Learning: Generative or Contrastive (2020, background): “On the other hand, achieving the same lower bound value can lead to vastly different representations and performance on downstream tasks, which indicates that it does not necessarily capture useful information of data [3] [131] [14].”
Direct Validation of the Information Bottleneck Principle for Deep Nets (2019, background): “This problem is mitigated if one resorts to stochastic DNNs via the introduction of noise after each layer, which ensures that I(X;Li) is finite [12, 3], as done in [2, 4, 8, 1, 5].”
ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding (2022, unknown): “We will also delve into models with both objective and subjective image quality remarkable, following exiting literature [10,18,38,43].”
High-Fidelity GAN Inversion for Image Attribute Editing (2021, methodology): “This conjecture is also supported by the Rate-Distortion theory [5,7,29], which will be reviewed in the Supplement.”
Source Coding of Audio Signals with a Generative Model (2020, background): “As also discussed in [23,24], there is a non-trivial tradeoff between preservation of a waveform match and preservation of the distribution of the signal.”
Lossy Compression for Lossless Prediction (2021, background): “Most research in neural compression is either focused on estimation and optimization of the rate term [81, 105–111] or on developing perceptually meaningful distortions [8, 10, 112, 113].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, methodology): “We hence use the LPIPS perceptual similarity metric [96] for measuring compression quality [7].”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “More recently, a rate-distortion-perception trade-off has been introduced in [40], in which the perceptual quality of an output image is defined as ‘the extent to which it is perceived as a valid (natural) sample’.”