Deep and Hierarchical Implicit Models

By Dustin Tran, Rajesh Ranganath, David M. Blei. 2017

101 citations recorded by the source corpus.

Implicit probabilistic models are a flexible class for modeling data. They define a process to simulate observations, and unlike traditional models, they do not require a tractable likelihood function. In this paper, we develop two families of models: hierarchical implicit models and deep implicit models. They combine the idea of implicit densities with hierarchical Bayesian modeling and deep neural networks. The use of implicit models with Bayesian analysis has been limited by our ability to perform accurate and scalable inference. We develop likelihood-free variational inference (LFVI). Key to LFVI is specifying a variational family that is also implicit. This matches the model's flexibility and allows for accurate approximation of the posterior. Our work scales up implicit models to sizes previously not possible and advances their modeling design. We demonstrate diverse applications: a large-scale physical simulator for predator-prey populations in ecology; a Bayesian generative adversarial network for discrete data; and a deep implicit model for text generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Spectral Normalization for Generative Adversarial Networks (2018, methodology): “This type of loss has been already proposed and used in Lim & Ye (2017); Tran et al. (2017).”
cGANs with Projection Discriminator (2018, methodology): “For the objective function, we used the following hinge version of the standard adversarial loss (1) (Lim & Ye, 2017; Tran et al., 2017) where the last activation function A of D is identity function. p ( z ) is standard Gaussian distribution and G ( z , y ) is the generator network.”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, methodology): “Speciﬁcally, we train with an alternating fashion using the hinge adversarial loss [42, 5, 53, 64]: We further introduce multi-level architecture search ( MLAS ) to AutoGAN, where the generator (and correspondingly the discriminator) will grow progressively.”
Boundary-Seeking Generative Adversarial Networks (2017, background): “Implicit inference There is a strong connection to implicit variational inference with GANs (Tran et al., 2017; Huszár, 2017), where the logit of a binary GAN (r in Equation 10) can be used as a proxy for the log-ratio in the variational lower bound.”
VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning (2017, background): “Likelihood-free variational inference (LFVI) [23] extends this idea to a hierarchical Bayesian setting.”
Variational Approaches for Auto-Encoding Generative Adversarial Networks (2017, background): “This trick is widespread [16, 18, 24, 25, 37], although perhaps not stated as explicitly.”
Consistency Regularization for Generative Adversarial Networks (2019, methodology): “Speciﬁcally, we evaluate regularization methods using three loss functions: the non-saturating loss (NS) (Goodfellow et al., 2014), the Wasserstein loss (WAS) (Arjovsky et al., 2017), and the hinge loss (Hinge) (Lim & Ye, 2017; Tran et al., 2017).”
Image Synthesis From Reconfigurable Layout and Style (2019, methodology): “4 The Loss Functions To train (ΘG,ΘD) in our LostGAN, we utilize the hinge version [32, 22] of the standard adversarial loss [6],”
Image Augmentations for GAN Training (2020, methodology): “We choose hinge loss [26, 36] for all the experiments.”
PixelGAN Autoencoders (2017, background): “The cost function of GAN is GANs can be considered within the wider framework of implicit generative models [2–4].”
An Overview of Deep Semi-Supervised Learning (2020, background): “Other formulations have been proposed to improve and stabilize the training procedure, such as the hinge-loss version of the adversarial loss [97, 149] and Wassertein GAN (WGAN) [4].”
Learning Latent Permutations with Gumbel-Sinkhorn Networks (2018, unknown): “However, this does not preclude inference: likelihood-free methods have recently been developed to enable learning in such implicitly deﬁned distributions (Ranganath et al., 2016; Tran et al., 2017).”
Improved Consistency Regularization for GANs (2020, background): “For example, the hinge loss on the discriminator (Lim & Ye, 2017; Tran et al., 2017) is deﬁned as: The Wassertein GAN (WGAN) (Arjovsky et al., 2017) is another successful reformulation which measures 1-Lipschitz constrained Wasserstein distance (Villani, 2008) between the target distribution and…”
Gradient Estimators for Implicit Models (2017, methodology): “Very recently, implicit distributions have also been considered as approximate posterior distributions for Bayesian inference, e.g. see [22, 47, 19, 17, 26, 13, 20, 43].”
Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions (2017, methodology): “For instance, recent implicit variational methods [18, 23, 16, 39] that utilise arbitrarily flexible parametric families for q(z;θ) require the computation of log density ratios [37], which is a very difficult problem in high dimensional settings.”
Implicit Variational Inference with Kernel Density Ratio Fitting (2017, background): “, 2014; Mohamed & Lakshminarayanan, 2016), there has been some works that extend the adversarial learning approach to the posterior inference of latent variable models (Mescheder et al., 2017; Huszár, 2017; Tran et al., 2017).”
Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference (2018, methodology): “, 2017), wild variational inference (Li & Liu, 2016), deep implicit models (Tran et al., 2017), implicit variational models (Huszár, 2017), and adversarial message passing approximations (Karaletsos, 2016).”
Unsupervised Object Segmentation by Redrawing (2019, methodology): “[35, 50] instead, and obtain the following formulation:”
Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis (2021, methodology): “In sum, we employ the hinge version of the adversarial loss (Lim & Ye (2017); Tran et al. (2017)) to iteratively train our D and G.”