Learning in Implicit Generative Models

By Shakir Mohamed, Balaji Lakshminarayanan. 2016

442 citations recorded by the source corpus.

Generative adversarial networks (GANs) provide an algorithmic framework for constructing generative models with several appealing properties: they do not require a likelihood function to be specified, only a generating procedure; they provide samples that are sharp and compelling; and they allow us to harness our knowledge of building highly accurate neural network classifiers. Here, we develop our understanding of GANs with the aim of forming a rich view of this growing area of machine learning---to build connections to the diverse set of statistical thinking on this topic, of which much can be gained by a mutual exchange of ideas. We frame GANs within the wider landscape of algorithms for learning in implicit generative models--models that only specify a stochastic procedure with which to generate data--and relate these ideas to modelling problems in related fields, such as econometrics and approximate Bayesian computation. We develop likelihood-free inference methods and highlight hypothesis testing as a principle for learning in implicit generative models, using which we are able to derive the objective function used by GANs, and many other related objectives. The testing viewpoint directs our focus to the general problem of density ratio estimation. There are four approaches for density ratio estimation, one of which is a solution using classifiers to distinguish real from generated data. Other approaches such as divergence minimisation and moment matching have also been explored in the GAN literature, and we synthesise these views to form an understanding in terms of the relationships between them and the wider literature, highlighting avenues for future exploration and cross-pollination.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Generative Models

Citations in the corpus

Spectral Normalization for Generative Adversarial Networks (2018, methodology): “This is a perspective that opens the door to the methods of implicit models (Mohamed & Lakshminarayanan, 2017; Tran et al., 2017) that can be used to carry out variational optimization without the direct knowledge of the density function.”
A-NICE-MC: Adversarial Training for MCMC (2017, background): “Notable examples include black-box variational inference and variational autoencoders [1–3], which enabled variational methods to benefit from the expressive power of deep neural networks, and adversarial training [4, 5], which allowed the training of new families of implicit generative models with ”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
Denoising Diffusion Implicit Models (2020, methodology): “DDIMs are implicit probabilistic models (Mohamed & Lakshminarayanan, 2016) and are closely related to DDPMs, in the sense that they are trained with the same objective function.”
Conditional Image Synthesis with Auxiliary Classifier GANs (2016, background): “Moreover, a theoretical understanding of GANs is an on-going research topic (Uehara et al., 2016; Mohamed & Lakshminarayanan, 2016).”
The frontier of simulation-based inference (2019, background): “First, there has been a significant cross-pollination between those studying simulationbased inference and those studying probabilistic models in machine learning (ML) (4), and the impressive growth of ML capabilities enables new approaches.”
Least Squares Generative Adversarial Networks (2016, background): “Another generalization work is presented in literature [28].”
Amortised MAP Inference for Image Super-resolution (2016, neutral): “In parallel to our contributions, theoretical work by Mohamed & Lakshminarayanan (2016) presented a unifying view on learning in GAN-style algorithms, of which our variant can be regarded a special case.”
Disentangling by Factorising (2018, methodology): “This new penalty is optimised using a discriminator network, following the divergence minimisation view of GANs [18, 20], and we show that our approach enhances disentanglement as well as reconstruction compared to the β-VAE.”
Troubling Trends in Machine Learning Scholarship (2018, background): “Seeking to resolve the confusion and provide historical context, [58] distinguishes between prescribed and implicit generative models.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “When considering these divergences, we see a connection between flowbased models, whose design principles use composition and change of variables, to the more general class of implicit probabilistic models (Diggle and Gratton, 1984; Mohamed and Lakshminarayanan, 2016).”
Revisiting Classifier Two-Sample Tests (2016, methodology): “), which tests for the difference in rank means of two samples; and the Kolmogorov-Smirnov tests ( ??”
Variational Approaches for Auto-Encoding Generative Adversarial Networks (2017, methodology): “While using class probability estimation is amongst the most popular, the density ratio can also be computed in several other ways including by f -divergence minimization and density-ratio matching [27, 34].”
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, methodology): “DDIMs use a generative Markov chain to reverse a non-Markovian inference process, and are a type of implicit probabilistic model (Mohamed and Lakshminarayanan, 2017).”
Distribution Matching in Variational Inference (2018, methodology): “Many GAN variants that use different losses have been introduced (Arjovsky et al., 2017; Mao et al., 2016; Mohamed and Lakshminarayanan, 2016; Nowozin et al., 2016; Zhao et al., 2017), but the adversarial game remains the same.”
Improved generator objectives for GANs (2016, neutral): “[7] discusses many approaches similar to the one presented here, but do not present experimental results.”
CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training (2017, methodology): “An implicit generative model [28] is a mechanism that can sample from a probability distribution but cannot provide likelihoods for data points.”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, methodology): “, using generative adversarial networks [8, 19] and variants.”