Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design

By Jonathan Ho, Xi Chen, A. Srinivas, Yan Duan, Pieter Abbeel. 2019

509 citations recorded by the source corpus.

Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models. Our implementation is available at this https URL

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “While diffusion models might resemble flows [9, 46, 10, 32, 5, 16, 23] and VAEs [33, 47, 37], diffusion models are designed so that q has no parameters and the top-level latent xT has nearly zero mutual information with the data x0.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “09 F LOW ++ [16] = 3 .”
Variational Diffusion Models (2021, methodology): “Figure 4: Our learned continuous-time variance-minimizing noise schedule SNR(t) for CIFAR-10, compared to its log-linear initialization and to schedules from the literature: [1] The -Linear schedule from Ho et al. [2020], [2] The ↵-Cosine schedule from Nichol and Dhariwal [2021]. All schedules were ”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “Diffusion models [24] have recently been shown to synthesize high quality images and audio [3, 10, 14, 27], which have long been led by other classes of generative models such as autoregressive models, GANs, VAEs, and flows [2, 5, 7, 9, 12, 13, 18, 29, 30, 32].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “…et al., 2016; Mittal et al., 2017; Marwah et al., 2017; Vahdat & Kautz, 2020; Child, 2020), Flows (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Ho et al., 2019a), Oord et al., 2017; Razavi et al., 2019; Ramesh et al., 2021), and lately diffusion and score matching models (Sohl-Dickstein et…”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “…2018) 3.35 - MintNet (Song et al., 2019b) 3.32 - Residual Flow (Chen et al., 2019) 3.28 46.37 FFJORD (Grathwohl et al., 2018) 3.40 - Flow++ (Ho et al., 2019) 3.29 - DDPM ( L ) (Ho et al., 2020) ď 3.70 * 13.51 DDPM ( L simple ) (Ho et al., 2020) ď 3.75 * 3.17 (Song & Ermon, 2019) 25.32 8.87…”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, methodology): “NVAE outperforms the state-of-the-art non-autoregressive flow and VAE models including IAF-VAE [4] and BIVA [36] on all the datasets, but ImageNet, in which NVAE comes second after Flow++[61].”
Improved Denoising Diffusion Probabilistic Models (2021, neutral): “(Ho et al., 2019) 3.69 3.08 PixelCNN (van den Oord et al., 2016c) 3.57 3.14 SPN (Menick & Kalchbrenner, 2018) 3.52 - NVAE (Vahdat & Kautz, 2020) - 2.91 Very Deep VAE (Child, 2020) 3.52 2.87 PixelSNAIL (Chen et al., 2018) 3.52 2.85 Image Transformer (Parmar et al., 2018) 3.48 2.90 Sparse Transformer…”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, methodology): “Currently, hierarchical VAEs attain theoretical code lengths comparable to NFs (Maaløe et al., 2019; Ho et al., 2019a) and DPMs achieve the best code lengths (Kingma et al., 2021) (2.49 bits/dim on CIFAR10, 3.72 bits/dim on ImageNet32 and 3.40 bits/dim on ImageNet64).”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “We apply variational dequantization (Ho et al., 2019) and variational data augmentation (Chen et al., 2020) to solve these problems.”
Neural Spline Flows (2019, methodology): “Flow++ [19] uses the CDF of a mixture of logistic distributions as a monotonic transformation in coupling layers, but requires bisection search to compute an inverse, since a closed form is not available.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “uniform in [0, 1] for the image example above), or learned, as for example in variational dequantization (Ho et al., 2019).”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, methodology): “CIFAR-10 ImageNet FFJORD [15] 3.40 - Flow++ [18] 3.08 3.86 Gated PixelCNN [35] 3.03 3.83 VFlow [4] 2.98 3.83 PixelCNN++ [40] 2.92 - NVAE [54] 2.91 3.92 Image Transformer [36] 2.90 3.77 Very Deep VAE [8] 2.87 3.80 PixelSNAIL [7] 2.85 3.80 δ -VAE [38] 2.83 3.77 Sparse Transformer [9] 2.80 - ScoreFlow…”
Residual Flows for Invertible Generative Modeling (2019, methodology): “In brackets are models that used “variational dequantization” (Ho et al., 2019), which we don’t compare against.”