Compression with Flows via Local Bits-Back Coding

By Jonathan Ho, Evan Lohn, Pieter Abbeel. 2019

63 citations recorded by the source corpus.

Likelihood-based generative models are the backbones of lossless compression due to the guaranteed existence of codes with lengths close to negative log likelihood. However, there is no guaranteed existence of computationally efficient codes that achieve these lengths, and coding algorithms must be hand-tailored to specific types of generative models to ensure computational efficiency. Such coding algorithms are known for autoregressive models and variational autoencoders, but not for general types of flow models. To fill in this gap, we introduce local bits-back coding, a new compression technique for flow models. We present efficient algorithms that instantiate our technique for many popular types of flows, and we demonstrate that our algorithms closely achieve theoretical codelengths for state-of-the-art flow models on high-dimensional data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Information Theory

Citations in the corpus

Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, methodology): “Currently, hierarchical VAEs attain theoretical code lengths comparable to NFs (Maaløe et al., 2019; Ho et al., 2019a) and DPMs achieve the best code lengths (Kingma et al., 2021) (2.49 bits/dim on CIFAR10, 3.72 bits/dim on ImageNet32 and 3.40 bits/dim on ImageNet64).”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, background): “Since better likelihoods are useful for applications including compression [21, 20, 51], semi-supervised learning [10], adversarial puriﬁcation [47], and comparing against likelihood-based generative models, we seek a training objective for score-based diffusion models that is as efﬁcient as score…”
Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks (2020, methodology): “Indeed, many of the deep generative models that are popular for unsupervised learning do have high parameter counts and are computationally expensive (Vaswani et al., 2017; Maaløe et al., 2019; Ho et al., 2019a).”
iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder (2021, methodology): “We use the implementations in [17] for evaluating rANS.”
Improving Inference for Neural Image Compression (2020, background): “Recent work has improved its efficiency in hierarchical latent variable models [Townsend et al., 2019b, Kingma et al., 2019], and extended it to flow models [Ho et al., 2019].”
iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression (2021, methodology): “LBB [13] is such an algorithm with large quantity of coding operations, and our re-implementation on LBB implies that coding takes about 90% of the compression latency.”
Latent Variable Models (2019, methodology): “On the other hand, fully observed models, as well as the flow based models of Hoogeboom et al. (2019) and Ho et al. (2019), do not require bits back coding, and therefore do not have to pay the one-off cost of starting a chain.”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, methodology): “At present it is unclear to what extent this affects LBB [18].”
HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models (2019, methodology): “5Local bits back, retrieved from Ho et al. (2019).”