Generative Models of Visually Grounded Imagination

By Ramakrishna Vedantam, Ian S. Fischer, Jonathan Huang, Kevin P. Murphy. 2017

154 citations recorded by the source corpus.

It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et.al. and the BiVCCA method of Wang et.al.) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Isolating Sources of Disentanglement in VAEs (2018, background): “The goal in this case is not perfect inversion but to distill simpler representation [15, 25, 26, 27, 28].”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, background): “There has also been work on learning disentangled representation without direct supervision [Higgins et al., 2018, Siddharth et al., 2017, Vedantam et al., 2018], some with sequential generative models [Eslami et al., 2016, Ba et al., 2015].”
Pros and Cons of GAN Evaluation Measures (2018, background): “[84] proposed a model for visually grounded imagination to create images of novel semantic concepts.”
Isolating Sources of Disentanglement in Variational Autoencoders (2018, background): “The goal in this case is not perfect inversion but to distill into a simpler representation (Hinton et al., 2011; Kulkarni et al., 2015; Cheung et al., 2014; Karaletsos et al., 2016; Vedantam et al., 2018).”
Small Sample Learning in Big Data Era (2018, background): “Likewise, Vedantam et al. (2018) firstly tried to define a visually grounded imagination with evaluation metrics of 3C’s, i.e. correctness, coverage, and compositionality, and further propose how to create generative models which can imagine compositionally novel concrete and abstract visual…”
Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018, unknown): “In summary, the contributions of this paper are as follows: First, we describe a product-of-experts scheme for deriving inference networks in a multimodal setting (similar to a recent proposal by (Vedantam et al., 2017)).”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, background): “However, the mapping from x to z is still missing so we need an additional embedding step [31] to learn the encoder qφx(z|x).”
Compositional Visual Generation with Energy Based Models (2020, methodology): “One approach decomposes data into disentangled factors of variation and situate each datapoint in the resulting - typically continuous - factor vector space [29, 9].”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, background): “We argue that these approaches are less than ideal, as they typically only target one of the proposed criteria (e.g. (one-way) cross-generation), often require additional modelling components and inference steps (JMVAE, TELBO, MFM), ignore the latent representation structures induced and largely onl”
Handling Incomplete Heterogeneous Data using VAEs (2018, background): “An alternative approach, proposed in (Vedantam et al., 2017), consists of exploiting the properties of Gaussian distributions in the linear factor analysis case (Williams and Nash, 2018) and extend them to nonlinear models, designing a factorized recognition model: q(zn|xn) = p(zn) ∏ d∈On q(zn|xnd),”
SCAN: Learning Hierarchical Compositional Visual Concepts (2017, methodology): “This is in contrast to JMVAE and TrELBO, which only produce meaningful sym2img samples if the value of the attribute is set to 1 (attribute is present) or 0 (attribute is not enforced).”
SCAN: Learning Abstract Hierarchical Compositional Visual Concepts (2017, background): “Bayesian models by (Tennenbaum, 1999; Lake et al., 2016) 2017) is similar to ours, it lacks the data efﬁciency and strong compositionality characteristic of our SCAN proposal.”
Improved Contrastive Divergence Training of Energy Based Models (2020, methodology): “We quantitatively compare compositional generations of our model with IGEBM and JVAE (Vedantam et al., 2018) models on the CelebA-HQ dataset.”