Densely connected normalizing flows

By Matej Grcic, Ivan Grubisic, Sinisa Segvic. 2021

66 citations recorded by the source corpus.

Normalizing flows are bijective mappings between inputs and latent representations with a fully factorized distribution. They are very attractive due to exact likelihood valuation and efficient sampling. However, their effective capacity is often insufficient since the bijectivity constraint limits the model width. We address this issue by incrementally padding intermediate representations with noise. We precondition the noise in accordance with previous invertible units, which we describe as cross-unit coupling. Our invertible glow-like modules increase the model expressivity by fusing a densely connected block with Nystrom self-attention. We refer to our architecture as DenseFlow since both cross-unit and intra-module couplings rely on dense connectivity. Experiments show significant improvements due to the proposed contributions and reveal state-of-the-art density estimation under moderate computing budgets.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Jun Zhu
2. Adji Bousso Dieng
3. Dan Friedman

In Computer Vision and Pattern Recognition

1. Jun Zhu
2. Chongxuan Li
3. Jianfei Chen
4. Dan Friedman
5. Il-Chul Moon
6. Wanmo Kang
7. Seungjae Shin
8. Siyu Wang

In Radiology, Nuclear Medicine and Imaging

1. Jun Zhu
2. Chongxuan Li
3. Adji Bousso Dieng
4. Dan Friedman
5. Dongjun Kim

Citations in the corpus

• Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, methodology): “The models are trained with PyTorch (Paszke et al., 2019) We use the down-sampled Ima-geNet datasets from https://image-net.org/data/ downsample/Imagenet32_train.zip , following Grci ´ c et al. (2021); Hazami et al. (2022).”
• Score Matching Model for Unbounded Data Score (2021, methodology): “The other direction is likelihood-based models, in which VAE [2], autoregressive models [3], and flow models [4] are involved.”
• The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, methodology): “…models we evaluate represent popular classes of generative models, including a variational autoencoder (vdvae; Child, 2020), a flow model (Dense-Flow; Grci´c et al., 2021), diffusion models (iddpm, Nichol and Dhariwal, 2021; adm Dhariwal and Nichol, 2021), gan-based models (Karras et al., 2019,…”