Categorical Reparameterization with Gumbel-Softmax
By Eric Jang, Shixiang Shane Gu, Ben Poole. 2016
6,514 citations recorded by the source corpus.
Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the inability to backpropagate through samples. In this work, we present an efficient gradient estimator that replaces the non-differentiable sample from a categorical distribution with a differentiable sample from a novel Gumbel-Softmax distribution. This distribution has the essential property that it can be smoothly annealed into a categorical distribution. We show that our Gumbel-Softmax estimator outperforms state-of-the-art gradient estimators on structured output prediction and unsupervised generative modeling tasks with categorical latent variables, and enables large speedups on semi-supervised classification.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Generative Models
- Stefano Ermon
- Yoshua Bengio
- Arash Vahdat
- Jiaming Song
- Diederik P. Kingma
- Aaron C. Courville
- Sergey Levine
- Jan Kautz
Citations in the corpus
- MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “Replacing the non-differentiable sampling operations with efficient gradient approximators (Jang et al., 2017)has not yet shown strong results with discrete GANs.”
- SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “tences explain their parent. Baziotis et al.(2019) performed sentence compression by chaining two sequence-to-sequence (seq2seq) models as an auto-encoder. A StraightThrough Gumbel-Softmax estimator (Jang et al., 2017) was used to sample an output sequence from the first seq2seq, which was encouraged”
- DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “Specifically, these works allow for sampling k-subsets from a categorical distribution, based on extensions or generalizations of the Gumbel-softmax trick [23, 15].”
- BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Gumbel-softmax relaxation (Jang et al., 2017; Maddison et al., 2017) is employed to train the model parameters.”
- Zero-Shot Text-to-Image Generation (2021, methodology): “We instead use the gumbel-softmax relaxation (Jang et al., 2016; Maddison et al., 2016), replacing the expectation over qφ with one over qτφ, where the relaxation becomes tight as the temperature τ → 0.”
- Neural Discrete Representation Learning (2017, background): “Recently a few authors have suggested the use of a new continuous reparemetrisation based on the so-called Concrete [25] or Gumbel-softmax [19] distribution, which is a continuous distribution and has a temperature constant that can be annealed during training to converge to a discrete distribution…”
- VQ-DRAW: A Sequential Discrete VAE (2020, background): “Several works have attempted to train discrete VAEs by using continuous latent variables that gradually sharpen during training [3] [4].”
- Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “One method to reparametrize categorical distributions is the Gumbel-Max trick (Gumbel, 1954), which has recently become popular in machine learning (Maddison et al., 2014; 2017; Jang et al., 2017; Kool et al., 2019).”
- Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “When the data are discrete, score function gradient estimators such as REINFORCE [66] or relaxed gradient estimators such as the Gumbel-Softmax [47, 31] can be used.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “The DALL-E [45] text-to-image model used dVAE, which uses gumbel-softmax [26, 39], relaxing the VQ-VAE’s online clustering.”
- CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “The standard reparameterization of categorical distribution is Gumbel-argmax [21], which means that zi×w+j = argmax k gk − ‖vk − Encφ(x)ij‖2/τ, gk ∼ Gumbel(0, 1), (5)”
- High-Resolution Complex Scene Synthesis with Transformers (2021, neutral): “In an orthogonal approach, [7] and [23] consider a more generic mechanism for conditional image synthesis via training an autoregressive (AR) transformer model in the latent space of a pre-trained discrete autoencoder model [31, 12], where [7] achieve efficient high-resolution synthesis by extending”
- Taming Transformers for High-Resolution Image Synthesis (2020, positive): “∗trained with Gumbel-Softmax reparameterization as in [59, 29].”
- NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
- Improved Training of Wasserstein GANs (2017, methodology): “Nonlinearity (G) [ReLU, LeakyReLU, softplus(2x+2) 2 1, tanh] Nonlinearity (D) [ReLU, LeakyReLU, softplus(2x+2) 2 1, tanh] Depth (G) [4, 8, 12, 20] Depth (D) [4, 8, 12, 20] Batch norm (G) [True, False] Batch norm (D; layer norm for WGAN-GP) [True, False] Base filter count (G) [32, 64, 128] Base filte”
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “Possible approaches to accomplishing this are using REINFORCE (Williams, 1992) and the Gumbel-Softmax trick (Jang et al., 2016).”
- Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “The authors model the gates using a modified version of the concrete distribution [32, 22], with parameters denoted by ϕ .”
- Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “To make the objective amenable to gradient-based optimization, we treat Ĝi,j and Ĝ a i,` as independent Bernoulli random variables with probabilities of success sigmoid(γz i,j) and sigmoid(γ a i,`) and optimize the continuous parameters γz and γa using the Gumbel-Softmax gradient estimator (Jang et ”
- Differentiable Causal Discovery from Interventional Data (2020, neutral): “Λ via the reparametrization trick [20, 29].”
- Adversarial Example Games (2020, methodology): “, adversarial traffic signs, we note that there is a significant gap in translating synthetic adversarial examples to adversarial examples that reside in the physical world [45].”