Variable Rate Image Compression with Recurrent Neural Networks

By George Toderici, Honglak Lee, Dragomir R. Radev, S. M. O'Malley, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, Rahul Sukthankar. 2015

829 citations recorded by the source corpus.

A large fraction of Internet traffic is now driven by requests from mobile devices with relatively small screens and often stringent bandwidth requirements. Due to these factors, it has become the norm for modern graphics-heavy websites to transmit low-resolution, low-bytecount image previews (thumbnails) as part of the initial page load process to improve apparent page responsiveness. Increasing thumbnail compression beyond the capabilities of existing codecs is therefore a current research focus, as any byte savings will significantly enhance the experience of mobile device users. Toward this end, we propose a general framework for variable-rate image compression and a novel architecture based on convolutional and deconvolutional LSTM recurrent networks. Our models address the main issues that have prevented autoencoder neural networks from competing with existing image compression algorithms: (1) our networks only need to be trained once (not per-image), regardless of input image dimensions and the desired compression rate; (2) our networks are progressive, meaning that the more bits are sent, the more accurate the image reconstruction; and (3) the proposed architecture is at least as efficient as a standard purpose-trained autoencoder for a given number of bits. On a large-scale benchmark of 32$\times$32 thumbnails, our LSTM-based approaches provide better visual quality than (headerless) JPEG, JPEG2000 and WebP, with a storage size that is reduced by 10% or more.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Can Active Memory Replace Attention? (2016, methodology): “They use convolutional LSTMs, an architecture similar to CGRU, and have recently been used for weather prediction [23] and image compression [24], in both cases surpassing the state-of-the-art.”
Neural GPUs Learn Algorithms (2015, methodology): “In image processing, convolutional LSTMs , an architecture similar to the Neural GPU, have recently been used for weather prediction (Shi et al., 2015) and image compression (Toderici et al., 2016).”
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “The gradients required for back-propagating through such a thresholding operation cannot be directly derived, so we use the straight-through estimation (STE) method [28,29].”
Adversarial Examples for Generative Models (2017, background): “While we do not speciﬁcally attack their models, viable compression schemes based on deep neural networks have already been proposed in the literature, showing promising results [21], [22].”
Memory Replay with Data Compression for Continual Learning (2022, positive): “On the other hand, neural compression approaches generally rely on optimizing Shannon’s rate-distortion trade-off, through RNNs (Toderici et al., 2015; 2017), auto-encoders (Agustsson et al.”
High-Fidelity Generative Image Compression (2020, methodology): “At the same time, deep learning-based lossy compression has seen great interest [45, 5, 31], where a neural network is directly optimized for the rate-distortion trade-off, which led to new state-of-the-art methods.”
Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations (2017, background): “There has been a surge of interest in DNN models for full-resolution image compression, most notably [32, 33, 4, 5, 30], all of which outperform JPEG [35] and some even JPEG 2000 [29] The pioneering work [32, 33] showed that progressive image compression can be learned with convolutional recurrent n”
Learned Video Compression (2018, background): “In the last two years, we have seen a great surge of ML-based image compression approaches [13, 39, 40, 5, 4, 12, 20, 38, 33, 18, 2, 22, 6, 3, 8, 27, 28].”
Lossy Image Compression with Compressive Autoencoders (2017, methodology): “We used the implementation of van der Walt et al. (2014) for SSIM and the implementation of Toderici et al. (2016b) for MS-SSIM.”
Real-Time Adaptive Image Compression (2017, background): “Toderici et al. (2015; 2016) explored various transformations for binary feature extraction based on different types of recurrent neural networks; the binary representations were then entropy-coded.”
Generative Adversarial Networks for Extreme Learned Image Compression (2018, methodology): “Some authors rely on advanced techniques including multi-scale decompositions [34], progressive encoding/decoding strategies [40, 41], and generalized divisive normalization (GDN) layers [5, 4].”
FVC: A New Framework towards Deep Video Compression in Feature Space (2021, methodology): “The RNN-based image compression methods [31, 32, 15] are ﬁrstly introduced to progressively compress images.”
Joint Autoregressive and Hierarchical Priors for Learned Image Compression (2018, methodology): “[30] who used a recurrent architecture based on LSTMs to learn multi-rate, progressive models.”
Soft-to-Hard Vector Quantization for End-to-End Learned Compression of Images and Neural Networks (2017, methodology): “The pioneering work of Toderici et al. (2015; 2016) showed that progressive image compression can be learned with convolutional recurrent neural networks (RNNs), employing a stochastic quantization method during training.”
Deep Learning for Joint Source-Channel Coding of Text (2018, methodology): “To convert h and c into binary vectors of length (cid:96)/ 2 we use the same technique as in [22, 23, 13].”
Deep Probabilistic Video Compression (2018, positive): “In Toderici et al. (2016; 2017), a LSTM based codec is used to capture spatial correlations of pixel values and can achieve different bit-rates without having to retrain the model.”
Autoencoder Based Image Compression: Can the Learning be Quantization Independent? (2018, positive): “Deep autoencoders have been shown as promising tools for finding alternative transforms [2, 3, 4].”
Improving Inference for Neural Image Compression (2020, background): “3), stochastic binarization [Toderici et al., 2016], and soft-to-hard quantization [Agustsson et al., 2017].”