Learning Implicit Generative Models with the Method of Learned Moments
By Suman V. Ravuri, Shakir Mohamed, Mihaela Rosca, Oriol Vinyals. 2018
30 citations recorded by the source corpus.
We propose a method of moments (MoM) algorithm for training large-scale implicit generative models. Moment estimation in this setting encounters two problems: it is often difficult to define the millions of moments needed to learn the model parameters, and it is hard to determine which properties are useful when specifying moments. To address the first issue, we introduce a moment network, and define the moments as the network's hidden units and the gradient of the network's output with the respect to its parameters. To tackle the second problem, we use asymptotic theory to highlight desiderata for moments -- namely they should minimize the asymptotic variance of estimated model parameters -- and introduce an objective to learn better moments. The sequence of objectives created by this Method of Learned Moments (MoLM) can train high-quality neural image samplers. On CIFAR-10, we demonstrate that MoLM-trained generators achieve significantly higher Inception Scores and lower Frechet Inception Distances than those trained with gradient penalty-regularized and spectrally-normalized adversarial objectives. These generators also achieve nearly perfect Multi-Scale Structural Similarity Scores on CelebA, and can create high-quality samples of 128x128 images.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- David M. Blei
- Yoshua Bengio
- Andriy Mnih
- Michalis K. Titsias
- Ruslan Salakhutdinov
- Mohit Bansal
- Li Fei-Fei
- Tie-Yan Liu
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Li Fei-Fei
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Jitendra Malik
- Bernt Schiele
- Ming-Hsuan Yang
Citations in the corpus
- Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, neutral): “4 MoLM (Ravuri et al., 2018) 7.”
- Generative Modeling by Estimating Gradients of the Data Distribution (2019, neutral): “4 MoLM [45] 7.”
- HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, methodology): “Even widely used metrics, such as Inception Score [52] and Fréchet Inception Distance [23], have been discredited for their application to non-ImageNet datasets [3, 48, 8, 46].”
- Prescribed Generative Adversarial Networks (2019, methodology): “…distance metrics and optimization methods (see, e.g., Li et al., 2015; Dziugaite et al., 2015; Nowozin et al., 2016; Arjovsky et al., 2017; Ravuri et al., 2018; Genevay et al., 2017) or using ideas from VAE s (Makhzani et al., 2015; Mescheder et al., 2017; Dumoulin et al., 2016; Donahue…”
- Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “, 2014) or a moment network in the method of moments (Ravuri et al., 2018).”
- Bridging the Gap Between $f$-GANs and Wasserstein GANs (2019, unknown): “, 2018), MoLM (Ravuri et al., 2018), SNGAN (Miyato et al.”
- Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator (2020, methodology): “We evaluated the following models as our baselines: we take PixelCNN [38], PixelIQN [29], and MoLM [30] as representatives of other types of generative models.”
- Recurrent machines for likelihood-free inference (2018, methodology): “the method of matching moments, whose principle is to check compatibility between the generated data and the observed data by looking at these moments (Ravuri et al., 2018).”