MintNet: Building Invertible Neural Networks with Masked Convolutions

By Yang Song, Chenlin Meng, Stefano Ermon. 2019

75 citations recorded by the source corpus.

We propose a new way of constructing invertible neural networks by combining simple building blocks with a novel set of composition rules. This leads to a rich set of invertible architectures, including those similar to ResNets. Inversion is achieved with a locally convergent iterative procedure that is parallelizable and very fast in practice. Additionally, the determinant of the Jacobian can be computed analytically and efficiently, enabling their generative use as flow models. To demonstrate their flexibility, we show that our invertible neural networks are competitive with ResNets on MNIST and CIFAR-10 classification. When trained as generative models, our invertible networks achieve competitive likelihoods on MNIST, CIFAR-10 and ImageNet 32x32, with bits per dimension of 0.98, 3.32 and 4.06 respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “The score of a distribution can be estimated by training a score-based model on samples with score matching (Hyv¨arinen, 2005; Song et al., 2019a .”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “Examples of flow-based models that use masking include IAF (Kingma et al., 2016), MAF (Papamakarios et al., 2017), NAF (Huang et al., 2018), B-NAF (De Cao et al., 2019), masked invertible network (MintNet) (Song et al., 2019) and masked convolutional generative flow (MaCow) (Ma et al., 2019).”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, background): “Even smaller models can be forged by combining with potentially more compact architectures, such as MintNet (Song et al., 2019).”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “…kernels can also be designed to to ensure tractability, such as via using 1 × 1 convolution (Kingma & Dhariwal, 2018), masking (Oord et al., 2016; van den Oord et al., 2016; Hoogeboom et al., 2019; Song et al., 2019; Ma et al., 2019) or imposing certain repeated structure (Karami et al., 2019).”
Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models (2021, background): “A large body of work is dedicated to building more expressive continuous normalizing ﬂows (Dinh et al., 2017; Germain et al., 2015; Kingma et al., 2016; Papamakarios et al., 2017; Chen et al., 2018; Song et al., 2019; Perugachi-Diaz et al., 2020).”
Why Normalizing Flows Fail to Detect Out-of-Distribution Data (2020, background): “We hypothesize that many of our conclusions about coupling layers extend at a high level to other types of normalizing ﬂows [e.g., 3, 6, 12, 20, 13, 30, 38, 17, 8].”
Semi-Supervised Learning with Normalizing Flows (2019, background): “Moreover, recent work (Dinh et al., 2016; Kingma & Dhariwal, 2018; Behrmann et al., 2018; Chen et al., 2019; Song et al., 2019) demonstrated that normalizing flows can produce high-fidelity samples for natural image datasets.”
Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving (2020, methodology): “MintNet (Song et al., 2019) proposes a ﬁxed-point iteration method based on Newton-Raphson to speed up the inversion of an autoregressive procedure, but it is limited to a particular model.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “A number of these apply variations on causal convolutions [229], including emerging convolutions [90] whose inverse is sequential, MaCow [144] which uses smaller conditional fields allowing more efficient sampling, and MintNet [203] which approximates the inverse using fixedpoint iteration.”
Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions (2021, background): “A large body of work is dedicated to building more expressive continuous normalizing ﬂows (Dinh et al., 2017; Germain et al., 2015; Kingma et al., 2016; Papamakarios et al., 2017; Chen et al., 2018; Song et al., 2019; Perugachi-Diaz et al., 2020).”