A RAD approach to deep mixture models

By Laurent Dinh, Jascha Narain Sohl-Dickstein, Razvan Pascanu, Hugo Larochelle. 2019

46 citations recorded by the source corpus.

Flow based models such as Real NVP are an extremely powerful approach to density estimation. However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables. This makes them poorly suited for modeling and representing discrete structures in data distributions, for example class membership or discrete symmetries. To address this difficulty, we present a normalizing flow architecture which relies on domain partitioning using locally invertible functions, and possesses both real and discrete valued latent variables. This Real and Discrete (RAD) approach retains the desirable normalizing flow properties of exact sampling, exact inference, and analytically computable probabilities, while at the same time allowing simultaneous modeling of both continuous and discrete structure in a data distribution.

Subfields: Artificial Intelligence, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “Such problems can be reduced by using mixtures of ﬂows (Dinh et al., 2019; Cornish et al., 2019) or augmenting the base space (Dupont et al., 2019) at the cost of limiting ex-pressibility or losing the exact density.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Additionally, the above formulation is the same as RAD (Dinh et al., 2019) discussed in section 5.”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, background): “If the inference process is deterministic yet non-invertible, one could learn to invert it stochastically (Dinh et al., 2019; Nielsen et al., 2020).”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “Another approach known as RAD [43] learns a partitioning of the data space into disjoint subsets fYigKi¼1, and applies piecewise bijections to each region gi : X !”
Normalizing Flows: An Introduction and Review of Current Methods (2020, methodology): “Dinh et al. [21] explore the idea that a coupling function does not need to be bijective, but just piecewise-bijective (Fig.”
The Power Spherical distribution (2020, background): “…ﬂows (Huang et al., 2018; De Cao et al., 2019) have been shown to be remarkably ﬂexible, they are still subject to topological constraints (Dinh et al., 2019; Cor-nish et al., 2019), which motivates extensions for complex manifolds (Brehmer & Cranmer, 2020; Wu et al., 2020) including…”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Normalizing flows [51, 192] are generative models that generate tractable distributions to model high-dimensional data [53, 120].”
Diffusion Normalizing Flow (2021, background): “Normalizing flows [8, 35] have recently received lots of attention due to its exact density evaluation and ability to model high dimensional data [20, 9].”
Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows (2019, methodology): “Our use of a continuous index overcomes several limitations associated with alternative approaches based on a discrete index (Dinh et al., 2019; Duan, 2019), which suffer either from a discontinuous loss landscape or an intractable computational complexity.”