GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution

By Matt J. Kusner, José Miguel Hernández-Lobato. 2016

341 citations recorded by the source corpus.

Generative Adversarial Networks (GAN) have limitations when the goal is to generate sequences of discrete elements. The reason for this is that samples from a distribution on discrete objects such as the multinomial are not differentiable with respect to the distribution parameters. This problem can be avoided by using the Gumbel-softmax distribution, which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function. In this work, we evaluate the performance of GANs based on recurrent neural networks with Gumbel-softmax output distributions in the task of generating sequences of discrete elements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Privacy-Preserving Generative Deep Neural Networks Support Clinical Data Sharing (2017, background): “New neural networks have been designed to deal more effectively with discrete data [19,20].”
Texygen: A Benchmarking Platform for Text Generation Models (2018, methodology): “Gumbel Softmax trick is a reparametrization trick used to replace the multinomial stochastic sampling in text generation [8].”
Toward Controlled Generation of Text (2017, background): “Applying discriminators to text generation is hard due to the non-differentiability of discrete samples (Yu et al., 2017; Zhang et al., 2016; Kusner & Hernndez-Lobato, 2016).”
Style Transfer from Non-Parallel Text by Cross-Alignment (2017, background): “, 2016), the training procedure can be effectively optimized through back-propagation (Kusner and Hernández-Lobato, 2016; Goyal et al., 2017).”
Synthesizing Tabular Data using Generative Adversarial Networks (2018, methodology): “However, generating discrete variables is a challenge for GANs: the authors in [26, 7] attempt a differential model by designing special functions or modifying the loss function; other researchers [50, 14, 46] use a reinforcement learning framework to train the non-differentiable model, making…”
A Distributional Approach to Controlled Text Generation (2020, methodology): “Continuous approximation using the Gumbel Softmax was developed for the training of Variational Autoencoders but several works have implemented it for natural language generation Shetty et al. (2017); Chu & Liu (2019); Kusner & Hernández-Lobato (2016).”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “In a similar line of work, we use generative adversarial networks (GANs) (Goodfellow et al., 2014) with the Gumbel-Softmax relaxation (Kusner and Hernández-Lobato, 2016; Nie et al., 2019b) and train a generator with GAN loss to imitate the creation of human-authored tricky, incorrect answer options.”
Adversarial Ranking for Language Generation (2017, background): “Secondly, most of the existing GANs assume the output of the discriminator to be a binary predicate indicating whether the given sentence is written by human or machine [4, 16, 18, 34, 35].”
Generating Multi-label Discrete Patient Records using Generative Adversarial Networks (2017, methodology): “Alternatively, one could use specialized distributions, such as the Gumbel-softmax (Jang et al., 2016; Kusner and Hernández-Lobato, 2016), a concrete distribution (Maddison et al.”
Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning (2020, methodology): “To address this problem, we take advantage of the Gumbel-Softmax trick (Jang et al., 2016; Kusner and Hernández-Lobato, 2016) to approximate the sampling operation.”
Generating Multi-label Discrete Electronic Health Records using Generative Adversarial Networks (2017, methodology): “Alternatively, one could use specialized distributions, such as the Gumbel-softmax (Jang et al., 2016; Kusner and Hernández-Lobato, 2016), a concrete distribution (Maddison et al.”
Adversarially Regularized Autoencoders (2017, methodology): “Recent work on generative adversarial networks (GANs) for text has mostly focused on dealing with the non-differentiable objective either through policy gradient methods (Che et al., 2017; Hjelm et al., 2018; Yu et al., 2017) or with the Gumbel-Softmax distribution (Kusner & Hernandez-Lobato, 2016).”
A Transformer-based Framework for Neutralizing and Reversing the Political Polarity of News Articles (2021, methodology): “We decide to use Gumbel Softmax [62] as a continuous alternative of the softmax function: where P ( 𝑥 𝑖 ) is the probability of certain token being selected in each step decoding, and 𝜏 is the annealing parameter used to control the approximation strength ( 𝜏 → 0 means Gumbel Softmax degenerates”
Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning (2019, methodology): “2000) or those that approximate the discrete tokens with temperature-softmax probability vectors (Kusner and Hernández-Lobato 2016; Zhang et al. 2017) are used.”