VFlow: More Expressive Generative Flows with Variational Data Augmentation

By Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, Tian Tian. 2020

65 citations recorded by the source corpus.

Generative flows are promising tractable models for density modeling that define probabilistic distributions with invertible transformations. However, tractability imposes architectural constraints on generative flows, making them less expressive than other types of generative models. In this work, we study a previously overlooked constraint that all the intermediate representations must have the same dimensionality with the original data due to invertibility, limiting the width of the network. We tackle this constraint by augmenting the data with some extra dimensions and jointly learning a generative flow for augmented data as well as the distribution of augmented dimensions under a variational inference framework. Our approach, VFlow, is a generalization of generative flows and therefore always performs better. Combining with existing generative flows, VFlow achieves a new state-of-the-art 2.98 bits per dimension on the CIFAR-10 dataset and is more compact than previous models to reach similar modeling quality.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, background): “However, previous works found that augmenting neural ODEs [16, 17] and more generally generative models [18, 19, 20, 21] with latent variables improves their expressivity.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, methodology): “38 Flow Models without any Autoregressive Components in the Generative Model VFlow [58] - 2.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…community, various likelihood-based generative models have been developed, including normalizing ﬂows (Dinh et al. (2017); Ho et al. (2019a); Chen et al. (2020b); Lu et al. (2021), NFs), variational auto-encoders (Kingma & Welling (2013); Rezende et al. (2014), VAEs), auto-regressive…”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “We apply variational dequantization (Ho et al., 2019) and variational data augmentation (Chen et al., 2020) to solve these problems.”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, neutral): “CIFAR-10 ImageNet FFJORD [15] 3.40 - Flow++ [18] 3.08 3.86 Gated PixelCNN [35] 3.03 3.83 VFlow [4] 2.98 3.83 PixelCNN++ [40] 2.92 - NVAE [54] 2.91 3.92 Image Transformer [36] 2.90 3.77 Very Deep VAE [8] 2.87 3.80 PixelSNAIL [7] 2.85 3.80 δ -VAE [38] 2.83 3.77 Sparse Transformer [9] 2.80 - ScoreFlow…”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “One approach to build a surjective layer is to augment the input space with additional dimensions allowing smoother transformation to be learned [24], [47], [94]; the inverse process, where some dimensions are factored out, is equivalent to a multi-scale architecture [42].”
Densely connected normalizing flows (2021, methodology): “We compare our model with VFlow [24] and NVAE [56] due to similar generative performance on CIFAR-10 and CelebA, respectively.”
Why Normalizing Flows Fail to Detect Out-of-Distribution Data (2020, background): “…and allows for eﬃcient Jacobian computation in (1): Flows with coupling layers Coupling layers can be stacked together into ﬂexible normalizing ﬂows: ◦ ◦ ◦ Examples of ﬂows with coupling layers include NICE [9], RealNVP [10], Glow [22], and many others [e.g., 4, 5, 11, 15, 16, 21, 25, 32].”
Implicit Normalizing Flows (2021, background): “Note that such approximation is rather different from the variational inference technique in Chen et al. (2020); Nielsen et al. (2020), because we only focus on the exact log density itself.”
iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder (2021, methodology): “In this paper, we focus on developing approaches with deep probabilistic models based on normalizing flows [11, 25, 32, 27, 7, 28, 23].”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “Furthermore, Chen et al. (2020) and Huang et al. (2020) recently trained augmented Normalizing Flows, which have conceptual similarities with our velocity augmentation.”