Latent Normalizing Flows for Discrete Sequences

By Zachary M. Ziegler, Alexander M. Rush. 2019

136 citations recorded by the source corpus.

Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discrete sequences poses significant additional challenges. We propose a VAE-based generative model which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space. In this setting, we find that it is crucial for the flow-based distribution to be highly multimodal. To capture this property, we propose several normalizing flow architectures to maximize model flexibility. Experiments consider common discrete sequence tasks of character-level language modeling and polyphonic music generation. Our results indicate that an autoregressive flow-based model can match the performance of a comparable autoregressive baseline, and a non-autoregressive flow-based model can improve generation speed with a penalty to performance.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “Rezende & Mohamed (2015), Chen et al. (2017) and Ziegler & Rush (2019) improve VAE performance by enhancing the expressive power of prior and posterior distribution with normalizing ﬂows.”
Neural Spline Flows (2019, background): “Non-linear squared flow [59] adds an inverse-quadratic perturbation to an affine transformation in an autoregressive flow, which is invertible under certain restrictions of the parameterization.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, neutral): “Ziegler and Rush (2019) use such an approach for character-level language modeling.”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, methodology): “Our model outperforms all nonautoregressive baselines except one, the Discrete Flow model [41] (for which unfortunately no open-source implementations exist), and is also faster than all but one method, the IAF/SCF model [49].”
Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions (2021, methodology): “8 IAF / SCF (cid:63) (Ziegler and Rush, 2019) 0.04 ± 0 .”
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (2019, unknown): “(2019)) and language modeling (Ziegler and Rush, 2019).”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, methodology): “In particular, they are IAF/SCF (Ziegler & Rush, 2019), the Autoregressive Argmax Flow (Hoogeboom et al., 2021), and the discrete flow (Tran et al., 2019) for autoregressive models.”
MaCow: Masked Convolutional Generative Flow (2019, background): “Ziegler and Rush (2019) adopted variational inference to apply generative ﬂows to discrete sequential data.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “malizing flows offer parallel synthesis, providing substantial speedup [179], [219], [264].”
Normalizing Flows: An Introduction and Review of Current Methods (2020, methodology): “Other approaches transform a discrete variable into a continuous latent variable with a variational autoencoder, and then apply normalizing flows in the continuous latent space [Wang and Wang, 2019; Ziegler and Rush, 2019].”
Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models (2021, methodology): “We utilize the ofﬁcial code repository from (Ziegler & Rush, 2019) in here 1 .”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, methodology): “els for text use LSTMs (Hochreiter and Schmidhuber, 1997) as their decoders (Yang et al., 2017; Ziegler and Rush, 2019; Ma et al., 2019).”
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (2022, background): “Several works (Kingma et al., 2016; Ziegler & Rush, 2019; Zhang et al., 2020) also explore normalizing flows on discrete data under the Variational Autoencoders (Kingma & Welling, 2013) framework by enriching the prior.”
Discrete Flows: Invertible Generative Models of Discrete Data (2019, methodology): “We follow the setup of Ziegler and Rush (2019), which to the best of our knowledge is the only comparable work with nonautoregressive language modeling.”