Emerging Convolutions for Generative Normalizing Flows

By Emiel Hoogeboom, Rianne van den Berg, Max Welling. 2019

105 citations recorded by the source corpus.

Generative flows are attractive because they admit exact likelihood optimization and efficient image synthesis. Recently, Kingma & Dhariwal (2018) demonstrated with Glow that generative flows are capable of generating high quality images. We generalize the 1 x 1 convolutions proposed in Glow to invertible d x d convolutions, which are more flexible since they operate on both channel and spatial axes. We propose two methods to produce invertible convolutions that have receptive fields identical to standard convolutions: Emerging convolutions are obtained by chaining specific autoregressive convolutions, and periodic convolutions are decoupled in the frequency domain. Our experiments show that the flexibility of d x d convolutions significantly improves the performance of generative flow models on galaxy images, CIFAR10 and ImageNet.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, background): “Flow-based generative models have received a lot of attention due to their advantages (Hoogeboom et al., 2019; Durkan et al., 2019; Serr`a et al., 2019).”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “To backpropagate through the discrete-valued functions round(·) and argmax, both Hoogeboom et al. (2019a) and Tran et al. (2019) use the straight-through gradient estimator (Bengio et al., 2013).”
WaveFlow: A Compact Flow-based Model for Raw Audio (2019, background): “Hoogeboom et al. (2019) generalizes the invertible convolution to operate on both channels and spatial axes.”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, background): “, 2019), and emerging convolutions (Hoogeboom et al., 2019).”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “…kernels can also be designed to to ensure tractability, such as via using 1 × 1 convolution (Kingma & Dhariwal, 2018), masking (Oord et al., 2016; van den Oord et al., 2016; Hoogeboom et al., 2019; Song et al., 2019; Ma et al., 2019) or imposing certain repeated structure (Karami et al., 2019).”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “A number of these apply variations on causal convolutions [229], including emerging convolutions [90] whose inverse is sequential, MaCow [144] which uses smaller con-”
MaCow: Masked Convolutional Generative Flow (2019, methodology): “, 2017) and Emerging Convolutions (Hoogeboom et al., 2019).”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “[111] have provided a more general solution for modelling d×d convolutions.”
Hamiltonian Generative Networks (2019, background): “…learning (Rezende & Mohamed, 2015; Kingma et al., 2016; Papamakarios et al., 2017; Dinh et al., 2017; Huang et al., 2018; Kingma & Dhariwal, 2018; Hoogeboom et al., 2019; Chen et al., 2019; Grathwohl et al., 2018), none of the approaches manage to produce both sampling and density evaluation…”
Cubic-Spline Flows (2019, unknown): “Several invertible transformations with a tractable Jacobian determinant have been proposed, including coupling transforms [4, 5], autoregressive transforms [3, 13, 17, 23], invertible convolutions [11, 16], transformations based on the matrixdeterminant lemma [27, 29], and continuous transformation”
Why Normalizing Flows Fail to Detect Out-of-Distribution Data (2020, background): “…and allows for eﬃcient Jacobian computation in (1): Flows with coupling layers Coupling layers can be stacked together into ﬂexible normalizing ﬂows: ◦ ◦ ◦ Examples of ﬂows with coupling layers include NICE [9], RealNVP [10], Glow [22], and many others [e.g., 4, 5, 11, 15, 16, 21, 25, 32].”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, background): “Examples include autoregressive neural networks [36, 42], variational autoencoders [26, 41, 45, 31] and normalizing flows [47, 46, 10, 41, 11, 37, 27, 21, 15, 14, 20, 17], with the latter two enjoying faster decoding times.”
Integer Discrete Flows and Lossless Compression (2019, methodology): “Flow-based generative models [7, 8, 22, 19, 14, 15] are advantageous over other generative models: i) they admit exact log-likelihood optimization in contrast with Variational AutoEncoders (VAEs) [18] and ii) drawing samples (and decoding) is comparable to inference in terms of computational cost, a”
Block Neural Autoregressive Flow (2019, methodology): “Orthogonal work has been done for constructing powerful invertible function such as invertible 1 × 1 convolution (Glow) by Kingma and Dhariwal (2018), invertible d × d convolutions (Hoogeboom et al., 2019), and invertible residual networks (Behrmann et al., 2018).”
Normalizing Flows: An Introduction and Review of Current Methods (2020, methodology): “Hoogeboom et al. [2019a] noted that fixing the permutation matrix limits the flexibility of the transformation, and proposed using the QR decomposition instead where the orthogonal matrix Q is described with Householder transforms.”
MintNet: Building Invertible Neural Networks with Masked Convolutions (2019, methodology): “Emerging convolutions [13] and MaCow [20] improve the Glow architecture by replacing 1 × 1 convolutions in the original Glow model with masked convolutions similar to those employed in MintNets.”