Towards Diverse and Natural Image Descriptions via a Conditional GAN

By Bo Dai, Sanja Fidler, Raquel Urtasun, Dahua Lin. 2017

480 citations recorded by the source corpus.

Despite the substantial progress in recent years, the image captioning techniques are still far from being perfect. Sentences produced by existing methods, e.g. those based on RNNs, are often overly rigid and lacking in variability. This issue is related to a learning principle widely used in practice, that is, to maximize the likelihood of training samples. This principle encourages high resemblance to the “ground-truth” captions, while suppressing other reasonable descriptions. Conventional evaluation metrics, e.g. BLEU and METEOR, also favor such restrictive methods. In this paper, we explore an alternative approach, with the aim to improve the naturalness and diversity – two essential properties of human expression. Specifically, we propose a new framework based on Conditional Generative Adversarial Networks (CGAN), which jointly learns a generator to produce descriptions conditioned on images and an evaluator to assess how well a description fits the visual content. It is noteworthy that training a sequence generator is nontrivial. We overcome the difficulty by Policy Gradient, a strategy stemming from Reinforcement Learning, which allows the generator to receive early feedback along the way. We tested our method on two large datasets, where it performed competitively against real people in our user study and outperformed other methods on various tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “To address this problem, we develop an instruction-trajectory compatibility model that outperforms all existing automatic evaluation metrics on instance-level evaluation without needing any reference instructions – making it suitable for use as a reward function in a reinforcement learning setting, ”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Reinforcement learning enables model optimization with non-differentiable evaluation metrics [14, 45, 61, 64].”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “Adversarial learning [6, 26] has been shown to be successful in dialog generation, translation, image captioning and a series of natural language generation tasks [7, 12, 27, 28, 29, 30, 31, 32, 33, 34].”
Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, background): “GANs are also very helpful for developing services for visually impaired people, such as images-to soundconverters using both GANs to generate descriptive texts from a given image [46] and another DL model to perform textto-speech conversion.”
Boundary-Seeking Generative Adversarial Networks (2017, methodology): “Many REINFORCE-based methods have been proposed for language modeling (Yu et al., 2016; Li et al., 2017; Dai et al., 2017) which are similar to our REINFORCE-based BGAN formulation and effectively use the sigmoid of the estimated loglikelihood ratio.”
Adversarial Ranking for Language Generation (2017, methodology): “rollouts methods [4, 35] to simulate intermediate rewards when a sequence is incomplete.”
Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, background): “Besides machine translation, there is work on latent variables for dialogue generation (Serban et al., 2017; Cao & Clark, 2017; Wen et al., 2017) and image captioning (Wang et al., 2017; Dai et al., 2017).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Dai et al. [26] also proposed a GAN based image captioning method.”
Guide Me: Interacting with Deep Networks (2018, background): “Other examples include generation of referring expressions [36, 44, 47, 62], segmentation or object retrieval from referring expressions [19, 29, 28], image captioning [16, 21, 33, 34, 48, 56] and visual dialog [18].”
Generative Adversarial Network in Medical Imaging: A Review (2018, background): “X-ray Dai et al. (2017b) D25, 26 adversarial loss is able to correct the shape inconsistency”
Feature-map-level Online Adversarial Knowledge Distillation (2020, methodology): “Adversarial training has been adapted to various tasks such as image translation (Isola et al., 2017; Zhu et al., 2017), captioning (Dai et al., 2017), semi-supervised learning (Miy-ato et al., 2016; Springenberg, 2015), reinforcement learning (Pfau & Vinyals, 2016), and many others.”
Latent Translation: Crossing Modalities by Bridging Generative Models (2019, background): “, 2018) including conditional generation (Mirza & Osindero, 2014), generation of one domain conditioned on another (Dai et al., 2017; Reed et al., 2016), generation of high-quality images (Karras et al.”
Measuring the Diversity of Automatic Image Descriptions (2018, methodology): “When we separate the compounds by length, we see that humans produce most compounds in any category, and the GAN-based systems (Dai et al., 2017; Shetty et al., 2017) produce more compounds of length 3 and 4 than the other systems. The system by Vinyals et al. (2017) also stands out in this regard.”
Audio-Visual Fusion for Sentiment Classiﬁcation using Cross-Modal Autoencoder (2018, background): “As a result, interesting and new problems such as visual question answering [11, 9, 12, 13], object discovery by multi-modal dialogue [9, 14] and text-to-image generation [15, 16, 17] have emerged.”
pix2code: Generating Code from a Graphical User Interface Screenshot (2017, background): “Generative Adversarial Networks (GANs) [5] have shown to be extremely powerful at generating images and sequences [2, 20].”
Inverse Cooking: Recipe Generation From Food Images (2018, methodology): “Following neural machine translation trends, autoregressive models have exhibited promising performance in image captioning [52, 59, 28, 20, 7, 46], where the goal is to provide a short description of the image contents, opening the doors to less constrained problems such as generating descriptive p”
Point Cloud GAN (2018, background): “Their wide range of applicability was also shown in many important problems, including data augmentation (Salimans et al., 2016), image style transformation (Zhu et al., 2017), image captioning (Dai et al., 2017) and art creations (Kang, 2017).”