Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables

By F. Kingma, Pieter Abbeel, Jonathan Ho. 2019

113 citations recorded by the source corpus.

The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-ANS), recently proposed by Townsend et al. (2019), makes bits-back coding practically feasible for latent variable models with one latent layer, but it is inefficient for hierarchical latent variable models. In this paper we propose Bit-Swap, a new compression scheme that generalizes BB-ANS and achieves strictly better compression rates for hierarchical latent variable models with Markov chain structure. Through experiments we verify that Bit-Swap results in lossless compression rates that are empirically superior to existing techniques. Our implementation is available at this https URL.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, background): “Practical versions for modern deep generative models were developed by Townsend et al. (2019); Kingma et al. (2019).”
iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression (2021, methodology): “Various lossless compression algorithms have been discovered based on different types of generative models [26, 35, 15, 3, 13, 36, 23, 27, 5].”
Latent Variable Models (2019, methodology): “The Bit-Swap technique (Kingma et al., 2019) requires that inference be done bottom up, and that generative and inference models must both be a Markov chain on z1, .”
Compression with Flows via Local Bits-Back Coding (2019, methodology): “If auxiliary bits are not available, they must be counted into the codelength for the first datapoint, which can make our method impractical when coding few datapoints or when no pre-transmitted random bits are present [42, 26].”
QARV: Quantization-Aware ResNet VAE for Lossy Image Compression (2023, methodology): “Existing methods have successfully applied VAEs, including a single-layer VAE [35] and hierarchical ones [2], [36], to loss-less compression of images.”
Integer Discrete Flows and Lossless Compression (2019, methodology): “The Bit-Swap results are retrieved from [23].”
Improving Inference for Neural Image Compression (2020, background): “Although lossy-bits-back coding in the present VAE only gave relatively minor benefits, it may reach its full potential in more hierarchical architectures as in [Kingma et al., 2019].”
Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks (2020, background): “…this is a useful result for any real-world application of these generative models. such as learned lossless compression (Townsend et al., 2019; Kingma et al., 2019; Townsend et al., 2020; Ho et al., 2019b; Hoogeboom et al., 2019), which could be made practical with these reduced memory and…”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, background): “Recent examples of such approaches are Local Bits-Back Coding with normalizing flows [18] and variational auto-encoders [26, 41] with bits-back coding such as Bits-Back with ANS [51], Bit-Swap [28] and HiLLoC [50].”
iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder (2021, background): “Alternatively, variational autoencoders (VAEs) [24, 26, 37] maximise a lower bound on the marginal data likelihood (otherwise known as the ELBO).”
HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models (2019, methodology): “The Bit-Swap technique (Kingma et al., 2019) requires that inference be done bottom up, and that generative and inference models must both be a Markov chain on z1, .”