Practical Lossless Compression with Latent Variables using Bits Back Coding

By James Townsend, Thomas Bird, David Barber. 2019

161 citations recorded by the source corpus.

Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present `Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational auto-encoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly amenable to parallelization, we conclude that with a sufficiently high quality generative model this scheme could be used to achieve substantial improvements in compression rate with acceptable running time. We make our implementation available open source at this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, background): “Practical versions for modern deep generative models were developed by Townsend et al. (2019); Kingma et al. (2019).”
Variational Diffusion Models (2021, methodology): “As a proof of concept of practical lossless compression, Table 2 reports net codelengths on the CIFAR10 test set for various settings of Teval using BB- ANS [Townsend et al., 2018], an implementation of bits-back coding based on asymmetric numeral systems [Duda, 2009].”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “With a model distribution close enough to the data distribution, many efﬁcient lossless neural compressors have been explored (Hoogeboom et al., 2019; van den Berg et al., 2020; Ho et al., 2019b; Townsend et al., 2019a; Kingma et al., 2021).”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, background): “Since better likelihoods are useful for applications including compression [21, 20, 51], semi-supervised learning [10], adversarial purification [47], and comparing against likelihood-based generative models, we seek a training objective for score-based diffusion models that is as efficient as score”
Compression with Flows via Local Bits-Back Coding (2019, methodology): “There is already a rich literature of tailored coding algorithms for autoregressive models and variational autoencoders built from conditional distributions which are already tractable for coding [38, 12, 18, 13, 42], but there are currently no such algorithms for general types of flow models [32].”
Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables (2019, methodology): “An option is to follow the discretization technique used in (Townsend et al., 2019) by dividing the latent space into bins that have equal mass under some distribution (as opposed to equal width).”
Latent Variable Models (2019, methodology): “Recently, zero-overhead bits back compression with a significantly simpler implementation has been developed by Townsend et al. (2019).”
Improving Inference for Neural Image Compression (2020, methodology): “Recent work has improved its efficiency in hierarchical latent variable models [Townsend et al., 2019b, Kingma et al., 2019], and extended it to flow models [Ho et al., 2019].”
iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder (2021, methodology): “With the bits-back coding framework [38], the theoretical codelength is exactly equal to the ELBO.”
Integer Discrete Flows and Lossless Compression (2019, methodology): “At this point, bits-back coding for VAEs has only been implemented for MNIST [32].”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, background): “Recent examples of such approaches are Local Bits-Back Coding with normalizing flows [18] and variational auto-encoders [26, 41] with bits-back coding such as Bits-Back with ANS [51], Bit-Swap [28] and HiLLoC [50].”
HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models (2019, methodology): “Recently, zero-overhead bits back compression with a significantly simpler implementation has been developed by Townsend et al. (2019).”
QARV: Quantization-Aware ResNet VAE for Lossy Image Compression (2023, methodology): “Existing methods have successfully applied VAEs, including a single-layer VAE [35] and hierarchical ones [2], [36], to loss-less compression of images.”