Discriminability Objective for Training Descriptive Captions

By Ruotian Luo, Brian L. Price, Scott D. Cohen, Gregory Shakhnarovich. 2018

211 citations recorded by the source corpus.

One property that remains lacking in image captions generated by contemporary methods is discriminability: being able to tell two images apart given the caption for one of them. We propose a way to improve this aspect of caption generation. By incorporating into the captioning training objective a loss component directly related to ability (by a machine) to disambiguate image/caption matches, we obtain systems that produce much more discriminative caption, according to human evaluation. Remarkably, our approach leads to improvement in other aspects of generated captions, reflected by a battery of standard scores such as BLEU, SPICE etc. Our approach is modular and can be applied to a variety of model/loss combinations commonly proposed for image captioning.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “Various methods have been delivered for image captioning (Karpa-thy and Fei-Fei, 2015; Rennie et al., 2017; Anderson et al., 2018a; Luo et al., 2018; Luo, 2020).”
Understanding and Evaluating Racial Biases in Image Captioning (2021, methodology): “We examine the captions generated by six image captioning models: (1) FC [59] is a simple sequence encoder that takes in image features encoded by a CNN; (2) Att2in [59] is similar but images are encoded using spatial features; (3) DiscCap [50] further adds a loss term to encourage discriminability;”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “Modern image captioning models employ an end-to-end encoder-decoder framework [10, 30, 29, 2, 27], i.”
Object Hallucination in Image Captioning (2018, methodology): “All models except for TopDown-BB, NBT, and GAN are implemented in the same open source framework from Luo et al. (2018).”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, background): “Self-retrieval for image captioning Prior works have proposed incorporating a self-retrieval loss into caption generation, with the intuition that good captions should be able to uniquely identify their images with high accuracy (Dai and Lin, 2017; Luo et al., 2018; Liu et al., 2018); monitoring thi”
Reasoning Visual Dialogs With Structural and Partial Observations (2019, background): “Another popular trend [15, 47, 24, 5, 42, 37, 6] in this area focuses on improving the discriminability of caption generations, such as stylized image captioning [15, 6], personalized image captioning [47], and context-aware image captioning [24, 5].”
Learning to Collocate Neural Modules for Image Captioning (2019, background): “Compared with them, modern image captioners which achieve superior performances are attention based encoder-decoder methods [53, 52, 43, 6, 60, 34, 2, 55, 35, 36, 56, 41, 12, 13].”
A Better Variant of Self-Critical Sequence Training (2020, background): “…proposed for image captioning task, SCST not only has become the new standard for training captioning models (Yao et al., 2017; Dognin et al., 2019; Luo et al., 2018; Jiang et al., 2018; Liu et al., 2018b; Ling and Fi-dler, 2017; Liu et al., 2018a; Chen et al., 2019; Yang et al., 2019; Zhao et…”
Diversifying Dialogue Generation with Non-Conversational Text (2020, result): “As has been shown in previous research, discriminative models are generally less biased to high-frequent utterances and more robust against their generative counterparts (Lu et al., 2017; Luo et al., 2018).”
On Hallucination and Predictive Uncertainty in Conditional Language Generation (2021, unknown): “Data-to-text generation (Kukich, 1983; McKeown, 1992) is a task to generate textual content conditioned on input content in the form of structured data such as tables.”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, methodology): “In [37, 33], a weight sum of CIDEr score and global image-text matching similarity score is used as the reward function for discriminative captions.”
Injecting Semantic Concepts into End-to-End Image Captioning (2021, background): “Existing efforts on image captioning are re-ﬂected from the perspective of novel architectures [11,53,81], more effective learning objectives [25,48,58], or large-scale VL pre-training [38,79,82], etc .”