Full Resolution Image Compression with Recurrent Neural Networks

By George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David C. Minnen, Joel Shor, Michele Covell. 2016

893 citations recorded by the source corpus.

This paper presents a set of full-resolution lossy image compression methods based on neural networks. Each of the architectures we describe can provide variable compression rates during deployment without requiring retraining of the network: each network need only be trained once. All of our architectures consist of a recurrent neural network (RNN)-based encoder and decoder, a binarizer, and a neural network for entropy coding. We compare RNN types (LSTM, associative LSTM) and introduce a new hybrid of GRU and ResNet. We also study one-shot versus additive reconstruction architectures and introduce a new scaled-additive framework. We compare to previous work, showing improvements of 4.3%–8.8% AUC (area under the rate-distortion curve), depending on the perceptual metric used. As far as we know, this is the first neural network architecture that is able to outperform JPEG at image compression across most bitrates on the rate-distortion curve on the Kodak dataset images, with and without the aid of entropy coding.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions (2017, background): “Recently deep neural networks (DNNs) have attained impressive performance in many fields such as image classification [1], semantic segmentation [2], and image compression [3].”
Conditional Image Synthesis with Auxiliary Classifier GANs (2016, background): “Improved image models advance the state-of-the-art in image denoising (Ball´e et al., 2015), compression (Toderici et al., 2016), in-painting (van den Oord et al., 2016a), and super-resolution (Ledig et al., 2016).”
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, background): “Standard quantitative measures such as PSNR and SSIM clearly fail to capture and accurately assess image quality with respect to the human visual system [47].”
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “For the STE model, we use convolutional gated recurrent units (Con-vGRU) [38] to downsample.”
Adversarial Examples for Generative Models (2017, background): “While we do not speciﬁcally attack their models, viable compression schemes based on deep neural networks have already been proposed in the literature, showing promising results [21], [22].”
Memory Replay with Data Compression for Continual Learning (2022, positive): “On the other hand, neural compression approaches generally rely on optimizing Shannon’s rate-distortion trade-off, through RNNs (Toderici et al., 2015; 2017), auto-encoders (Agustsson et al., 2017) and GANs (Mentzer et al., 2020).”
Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations (2017, methodology): “Concurrent to our work, the method of [16] builds on the architecture proposed in [33], and shows that impressive performance in terms of the MS-SSIM metric can be obtained by incorporating it into the optimization (instead of just minimizing the MSE).”
End-To-End Optimized Speech Coding with Deep Neural Networks (2017, methodology): “S D ] 8 J ul 2 02 1 borious domain-specific research; industry standards such as MP3, JPEG, and AMR-WB took years to develop and were largely hand-designed.”
Learned Video Compression (2018, background): “In the last two years, we have seen a great surge of ML-based image compression approaches [13, 39, 40, 5, 4, 12, 20, 38, 33, 18, 2, 22, 6, 3, 8, 27, 28].”
Lossy Image Compression with Compressive Autoencoders (2017, methodology): “We used the implementation of van der Walt et al. (2014) for SSIM and the implementation of Toderici et al. (2016b) for MS-SSIM.”
Real-Time Adaptive Image Compression (2017, background): “Toderici et al. (2015; 2016) explored various transformations for binary feature extraction based on different types of recurrent neural networks; the binary representations were then entropy-coded.”
Generative Adversarial Networks for Extreme Learned Image Compression (2018, methodology): “The so-obtained model has a bitrate of 0.07 bpp and gets a slightly better MS-SSIM than BPG at the same bpp on the validation set.”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, methodology): “The RNN-based image compression methods [31, 32, 15] are firstly introduced to progressively compress images.”
Joint Autoregressive and Hierarchical Priors for Learned Image Compression (2018, methodology): “To improve compression performance, recent methods have focused on better encoder/decoder transforms and on more sophisticated entropy models [4]–[14].”
Generative Compression (2017, background): “These samples also appear sharper, with fewer artifacts compared to [7], [8] approach set to maximum compression.”
Deep Generative Models for Distribution-Preserving Lossy Compression (2018, background): “Compressive autoencoders [3, 5, 6, 8, 13, 10] and recurrent neural networks (RNNs) [1, 2, 7] have emerged as the most popular DNN architectures for compression.”
Variable Rate Deep Image Compression With a Conditional Autoencoder (2019, methodology): “We also note that variable-rate models using recurrent neural networks (RNNs) were proposed in [9, 24].”