Discrete Flows: Invertible Generative Models of Discrete Data

By Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Laurent Dinh, Ben Poole. 2019

133 citations recorded by the source corpus.

While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We consider two flow architectures: discrete autoregressive flows that enable bidirectionality, allowing, for example, tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows that enable efficient non-autoregressive generation as in RealNVP. Empirically, we find that discrete autoregressive flows outperform autoregressive baselines on synthetic discrete distributions, an addition task, and Potts models; and bipartite flows can obtain competitive performance with autoregressive baselines on character-level language modeling for Penn Tree Bank and text8.

Subfields: Artificial Intelligence, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

Citations in the corpus

Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Recently, ﬂow-based models (henceforth simply called ﬂow models ) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Tran et al. (2019) take this approach, showing performance in character-level language modeling competitive to RNNs while having superior generation runtime.”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, methodology): “Model Model steps NLL (bits/char) (#) Sample time (s) (#) Discrete Flow [41] (8 ⇥ 3 layers) 1.”
Normalizing Flows: An Introduction and Review of Current Methods (2020, background): “[94] model bijective functions on a finite set and show that, in this case, the change of variables is given by the formula: pYðyÞ 1⁄4 pZðg 1ðyÞÞ, i.”
Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions (2021, methodology): “Final remarks We have had extensive contact with the authors of (Tran et al., 2019) to resolve this issue over the course of several months.”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, methodology): “Tran et al. [52] also introduce an autoregressive version, where In practice, each element x i , s i and t i for i ∈ { 1 , ..., d } is represented as a one-hot vector.”
Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, background): “Flow-based models have gained popularity in generative modeling [9, 10, 31, 18, 3, 35, 53] and variational inference [30, 49, 32, 28, 26].”
Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models (2021, background): “Tran et al. (2019) propose invertible transformations for categorical variables directly.”
A RAD approach to deep mixture models (2019, background): “Similar to Rolfe (2017), the learning of discrete variables is achieved by relying on the the continuous component of the model, and, as opposed to other approaches (Bengio et al., 2013; Raiko et al., 2015; Jang et al., 2017; Maddison et al., 2017; Grathwohl et al., 2018; Tucker et al., 2017; Tran e”
Simplified and Generalized Masked Diffusion for Discrete Data (2024, unknown): “1.39 Discrete Flow [59] 1.23 Transformer AR [14] 1.23”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, methodology): “In particular, they are IAF/SCF (Ziegler & Rush, 2019), the Autoregressive Argmax Flow (Hoogeboom et al., 2021), and the discrete flow (Tran et al., 2019) for autoregressive models.”
Integer Discrete Flows and Lossless Compression (2019, background): “Around time of submission, we found unpublished concurrent work [33] that explores discrete flows.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “Integer discrete flows (IDF) achieve this using additive coupling layers, rounding translation values to the nearest integer and approximating gradients with the straight-through estimator [93]; discrete flows [219] apply affine coupling layers in modulo space”
Variational Causal Networks: Approximate Bayesian Inference over Causal Structures (2021, methodology): “be obtained by combining with normalising flows for discrete variables [35, 18].”
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (2022, background): “Discrete ﬂows (Tran et al., 2019; Hoogeboom et al., 2019; Lindt & Hoogeboom, 2021) introduce invertible transformations of random variables in discrete space without the need of computing the log-determinant of Jacobian.”