Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

By Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, Luc Van Gool. 2017

535 citations recorded by the source corpus.

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Efficiency & Compression

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Neural Compression [9], [297], [435] instead expliots generative models to optimize the storage overhead of input data without much distortion, which can benefit both efficacy”
Memory Replay with Data Compression for Continual Learning (2022, neutral): “On the other hand, neural compression approaches generally rely on optimizing Shannon’s rate-distortion trade-off, through RNNs (Toderici et al., 2015; 2017), auto-encoders (Agustsson et al., 2017) and GANs (Mentzer et al., 2020).”
High-Fidelity Generative Image Compression (2020, methodology): “Initial works relied on RNNs [45, 47], while subsequent works were based on auto-encoders [5, 44, 1].”
Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding (2019, methodology): “The original dimension S is recovered in the decoder with recently proposed sub-pixel convolution [25], which forms the upsampling operation.”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Many other works explore learning (variational) autoencoders with (vector-)quantized latent representation with a focus on generative modeling [62, 63, 59, 60] and compression [64], rather than representation learning.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “In particular, the work of [1, 30, 74, 83, 116, 166, 176, 184, 248] clusters the weights into different groups and use the centroid of each group as quantized values during inference.”
Generative Adversarial Networks for Extreme Learned Image Compression (2018, methodology): “The average number of bits needed to encode ŵ is measured by the entropy H(ŵ), which can be modeled with a prior [1] or a conditional probability model [30].”
Video Compression through Image Interpolation (2018, background): “Agustsson et al. [4] use soft assignment to approximate quantization.”
Practical Full Resolution Learned Lossless Image Compression (2018, background): “While it is known that likelihood-based discrete generative models can in theory be used for lossless compression [37], recent work on learned compression using deep neural networks has solely focused on lossy compression [4, 38, 39, 32, 1, 3].”
Harp-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding (2021, unknown): “Since Q is not a differentiable process, we employ soft-to-hard quantization [17], which softens the bin assignment during training.”
Deep Generative Models for Distribution-Preserving Lossy Compression (2018, background): “Compressive autoencoders [3, 5, 6, 8, 13, 10] and recurrent neural networks (RNNs) [1, 2, 7] have emerged as the most popular DNN architectures for compression.”
Joint Autoregressive and Hierarchical Priors for Learned Image Compression (2018, background): “Many learning-based methods take the form of an autoencoder, and separate models are trained to target different bit rates instead of training a single recurrent model [5]–[7], [9], [11], [12], [14], [34], [35].”
Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding (2020, methodology): “As detailed in [24], basic information theory calculates H(h) as −k p(βk) log2 p(βk), where p(βk) denotes the occurrence probability of the k-th cluster defined in the softto-hard quantization.”
Learning to Cooperate in Decentralized Wireless Networks (2018, methodology): “We adopt the approach used in [14] for message quantization in our paper.”
Scale-Space Flow for End-to-End Optimized Video Compression (2020, methodology): “Image Compression Research on learning-based image compression [7, 10, 32, 4, 29, 8, 25, 5] has shown significant progress in terms of rate–distortion performance compared to standard codecs such as JPEG [34], JPEG2000 [21] and BPG [11].”
Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules (2020, background): “In the early stage, some works are proposed to deal with non-differential quantization and rate estimation to make end-to-end training possible, such as [6, 7, 8].”