Variational Autoencoder for Deep Learning of Images, Labels and Captions

By Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, Lawrence Carin. 2016

851 citations recorded by the source corpus.

A novel variational autoencoder is developed to model images, as well as associated labels or captions. The Deep Generative Deconvolutional Network (DGDN) is used as a decoder of the latent image features, and a deep Convolutional Neural Network (CNN) is used as an image encoder; the CNN is used to approximate a distribution for the latent DGDN features/code. The latent code is also linked to generative models for labels (Bayesian support vector machine) or captions (recurrent neural network). When predicting a label/caption for a new image at test, averaging is performed across the distribution of latent codes; this is computationally efficient as a consequence of the learned CNN-based encoder. Since the framework is capable of modeling the image in the presence/absence of associated labels/captions, a new semi-supervised setting is manifested for CNN learning with images; the framework even allows unsupervised CNN learning, based on images alone.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “We follow standard practice [38, 30] and perform experiments where class-balanced labels are available for only 10% of the dataset.”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, background): “VAEs [32], [33] jointly train an encoder and decoder where the former maps images into latent distribution and the latter generates images based on the latent distribution.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, neutral): “, “transductive” models [14, 22, 21], graph-based methods [49, 4, 29], generative modeling [3, 27, 41, 9, 17, 23, 38, 34, 42], etc.”
Adversarial Feature Matching for Text Generation (2017, background): “It would be also interesting to generate text by conditioning on observed images (Pu et al., 2016).”
Semantic Compositional Networks for Visual Captioning (2016, neutral): “Moreover, a variational autoencoder was developed in [34] for image captioning.”
Deconvolutional Latent-Variable Model for Text Sequence Matching (2017, background): “So motivated, we propose to infer the embedding for each sentence with deep generative models, due to their ability to make effective use of unlabeled data and learn abstract features from complex data (Kingma et al. 2014; Yang et al. 2017; Pu et al. 2016; Wang et al. 2017).”
Deconvolutional Paragraph Representation Learning (2017, background): “Alternatively, one can learn the autoencoder and classifier jointly, by specifying a classification model whose input is the latent representation, h; see for instance [32, 33].”
Plant diseases and pests detection based on deep learning: a review (2021, background): “In addition, Generative Adversarial Networks (GANs) [93] and Variational automatic encoder (VAE) [94] can generate more diverse samples to enrich limited datasets.”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, methodology): “This is commonly seen in conditional VAE methods for attribute → image or image → caption generation (Pandey and Dukkipati, 2017; Pu et al., 2016; Sohn et al., 2015; Yan et al., 2016), as well as generative adversarial network (GAN)-based models for cross domain image-to-image translation (Ledig et…”
An Interpretable Generative Model for Handwritten Digits Synthesis (2019, background): “It is desired to provide explanable models for both VAEs and GANs.”
Deep Collaborative Embedding for Social Image Understanding (2019, methodology): “Many methods have been proposed to explore images and the associated text information [9], [12], [13], [14], [15], [16], [17], [18], [19], [20].”
Recurrent Highway Networks with Language CNN for Image Captioning (2016, methodology): “[28] use CNN as an image encoder and use a deep generative deconvolutional network as a decoder, together with a gated recurrent unit to generate caption.”
An Introduction to Variational Autoencoders (2019, positive): “The application to semi-supervised learning has been studied in (Kingma et al., 2014; Pu et al., 2016; Xu et al., 2017) among other work.”
Widar3.0: Zero-Effort Cross-Domain Gesture Recognition With Wi-Fi (2021, methodology): “For methods 3) and 4), we build a VAE model [65] with three fully-connected layers as Encoder and another three fully-connected layers as Decoder, which have sizes of {1024, 512, 128} and {128, 512, 1024} respectively.”
SCAN: Learning Abstract Hierarchical Compositional Visual Concepts (2017, background): “…Sohn et al., 2015; Pandey & Dukkipati, 2017) ; and 3) multimodal generative models that aim to embed visual and symbolic inputs in a joint latent space in order to be able to run bi-directional inferences (Srivastava & Salakhutdinov, 2014; Suzuki et al., 2017; Pu et al., 2016; Wang et al., 2016).”
JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets (2018, background): “There are also many recent works concerned with integrating variational autoencoder (VAE) (Kingma & Welling, 2013; Pu et al., 2016) and GAN concepts for improved data generation and latent code inference (Hu et al., 2017).”
Triangle Generative Adversarial Networks (2017, result): “aining of -GAN. Instead of using adversarial loss, there also exists work that uses supervised learning [27] for joint-distribution matching, and variational autoencoders for semi-supervised learning [28, 29]. Lastly, our work is also closely related to the recent work of [30, 31, 32], which treats ”