Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning

By Jyoti Aneja, Harsh Agrawal, Dhruv Batra, Alexander Gerhard Schwing. 2019

72 citations recorded by the source corpus.

Diverse and accurate vision+language modeling is an important goal to retain creative freedom and maintain user engagement. However, adequately capturing the intricacies of diversity in language models is challenging. Recent works commonly resort to latent variable models augmented with more or less supervision from object detectors or part-of-speech tags. In common to all those methods is the fact that the latent variable either only initializes the sentence generation process or is identical across the steps of generation. Both methods offer no fine-grained control. To address this concern, we propose Seq-CVAE which learns a latent space for every word. We encourage this temporal latent space to capture the 'intention' about how to complete the sentence by mimicking a representation which summarizes the future. We illustrate the efficacy of the proposed approach on the challenging MSCOCO dataset, significantly improving diversity metrics compared to baselines while performing on par w.r.t. sentence quality.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Generative models trained on large amounts of unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67].”
Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue (2020, methodology): “We here use the GRU (Li et al., 2017; Aneja et al., 2019) to sequentially condition previously selected knowledge to π θ and q φ .”
From Show to Tell: A Survey on Image Captioning (2021, background): “…some attempts to encourage naturalness and diversity with conditional GANs [148], contrastive learning [192], variational auto-encoders [194], part-of-speech tagging [198], or word latent spaces [195], further research is needed to design models that are suitable for real-world applications.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “However, these solutions tend to underperform in terms of caption quality, which is partially recovered by using variational auto-encoders [216], [217], [218], [219].”
Say As You Wish: Fine-Grained Control of Image Caption Generation With Abstract Scene Graphs (2020, methodology): “For the diversity measurement, we sample the same number of image captions for each model, and evaluate the diversity of sampled captions using two types of metrics: 1) n-gram diversity (Div-n): a widely used metric [4, 7] which is the ratio of distinct n-grams to the total number of words in the be”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, background): “…2019; Razavi et al., 2019), music synthesis (Dhariwal et al., 2020), speech generation (Oord et al., 2016; Ping et al., 2020), image captioning (Aneja et al., 2019; Deshpande et al., 2019; Aneja et al., 2018), semi-supervised learning (Kingma et al., 2014; Izmailov et al., 2020), and…”