The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
By Chris J. Maddison, Andriy Mnih, Yee Whye Teh. 2016
2,946 citations recorded by the source corpus.
The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propagated by the chain rule through the graph are low variance unbiased estimators of the gradients of the expected loss. While many continuous random variables have such reparameterizations, discrete random variables lack useful reparameterizations due to the discontinuous nature of discrete states. In this work we introduce Concrete random variables---continuous relaxations of discrete random variables. The Concrete distribution is a new family of distributions with closed form densities and a simple reparameterization. Whenever a discrete stochastic node of a computation graph can be refactored into a one-hot bit representation that is treated continuously, Concrete stochastic nodes can be used with automatic differentiation to produce low-variance biased gradients of objectives (including objectives that depend on the log-probability of latent stochastic nodes) on the corresponding discrete graph. We demonstrate the effectiveness of Concrete relaxations on density estimation and structured prediction tasks using neural networks.
Subfields: Safety, Risk, Reliability and Quality, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Safety, Risk, Reliability and Quality
- Max Welling
- Babak Ehteshami Bejnordi
- Stefano Ermon
- Tijmen Blankevoort
- Arash Vahdat
- Ian J. Goodfellow
- Adversarial Examples
- J. Scott McCarley
In Generative Models
- Stefano Ermon
- Yoshua Bengio
- Arash Vahdat
- Aaron C. Courville
- Diederik P. Kingma
- Max Welling
- Jan Kautz
- Jiaming Song
Citations in the corpus
- DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “Specifically, these works allow for sampling k-subsets from a categorical distribution, based on extensions or generalizations of the Gumbel-softmax trick [23, 15].”
- BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Gumbel-softmax relaxation (Jang et al., 2017; Maddison et al., 2017) is employed to train the model parameters.”
- Zero-Shot Text-to-Image Generation (2021, methodology): “We instead use the gumbel-softmax relaxation (Jang et al., 2016; Maddison et al., 2016), replacing the expectation over qφ with one over qτφ, where the relaxation becomes tight as the temperature τ → 0.”
- Neural Discrete Representation Learning (2017, background): “Recently a few authors have suggested the use of a new continuous reparemetrisation based on the so-called Concrete [25] or Gumbel-softmax [19] distribution, which is a continuous distribution and has a temperature constant that can be annealed during training to converge to a discrete distribution…”
- VQ-DRAW: A Sequential Discrete VAE (2020, background): “Several works have attempted to train discrete VAEs by using continuous latent variables that gradually sharpen during training [3] [4].”
- Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “One method to reparametrize categorical distributions is the Gumbel-Max trick (Gumbel, 1954), which has recently become popular in machine learning (Maddison et al., 2014; 2017; Jang et al., 2017; Kool et al., 2019).”
- Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “When the data are discrete, score function gradient estimators such as REINFORCE [66] or relaxed gradient estimators such as the Gumbel-Softmax [47, 31] can be used.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “The DALL-E [45] text-to-image model used dVAE, which uses gumbel-softmax [26, 39], relaxing the VQ-VAE’s online clustering.”
- NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
- Improved Training of Wasserstein GANs (2017, methodology): “Other attempts at language modeling with GANs [31, 14, 29, 5, 15, 10] typically use discrete models and gradient estimators [27, 12, 16].”
- Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “The authors model the gates using a modified version of the concrete distribution [32, 22], with parameters denoted by ϕ .”
- Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “To make the objective amenable to gradient-based optimization, we treat Ĝi,j and Ĝ a i,` as independent Bernoulli random variables with probabilities of success sigmoid(γz i,j) and sigmoid(γ a i,`) and optimize the continuous parameters γz and γa using the Gumbel-Softmax gradient estimator (Jang et ”
- Differentiable Causal Discovery from Interventional Data (2020, methodology): “Λ via the reparametrization trick [20, 29].”
- Adversarial Example Games (2020, methodology): “The gradient is backprogated across this categorical variable using the gumble-softmax trick [45, 52].”
- Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “For discrete random variables, future work can study using the concrete relaxation [42] or the family of REINFORCE-based techniques [65, 46, 60].”
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “Intuitively, the Hard Concrete distribution is obtained by stretching the binary version of the Concrete (aka Gumbel softmax) distribution (Maddison et al., 2017; Jang et al., 2017) from the original support of (0 , 1) to ( − (cid:15), 1 + (cid:15) ) and then collapsing the probability mass…”
- Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “As in a standard VAE [37], the categorical VAE (CatVAE) [30, 31] optimises a lower bound for log p θ ( y 1: T ) using an encoder q φ ( z | x 1: T ) to approximate the intractable posterior p θ ( z | x 1: T ) , with φ and θ denoting the encoder and decoder weights, respectively.”
- Categorical Reparameterization with Gumbel-Softmax (2016, background): “…approximation to arg max , and generate k -dimensional sample vectors y ∈ ∆ k − 1 where The density of the Gumbel-Softmax distribution (derived in Appendix B) is: This distribution was independently discovered by Maddison et al. (2016), where it is referred to as the concrete distribution.”
- NIPS 2016 Tutorial: Generative Adversarial Networks (2016, methodology): “Using the concrete distribution (Maddison et al., 2016) or Gumbel-softmax (Jang et al.”
- Style Transfer from Non-Parallel Text by Cross-Alignment (2017, methodology): “continuous relaxation to approximate the discrete sampling process (Jang et al., 2016; Maddison et al., 2016), the training procedure can be effectively optimized through back-propagation (Kusner and Hernández-Lobato, 2016; Goyal et al.”