Context-Aware Captions from Context-Agnostic Supervision

By Ramakrishna Vedantam, Samy Bengio, Kevin P. Murphy, Devi Parikh, Gal Chechik. 2017

153 citations recorded by the source corpus.

We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation). For example, given images and captions of siamese cat and tiger cat, we generate language that describes the siamese cat in a way that distinguishes it from tiger cat. Our key novelty is that we show how to do joint inference over a language model that is context-agnostic and a listener which distinguishes closely-related concepts. We first apply our technique to a justification task, namely to describe why an image contains a particular fine-grained category as opposed to another closely-related category of the CUB-200-2011 dataset. We then study discriminative image captioning to generate language that uniquely refers to one of two semantically-similar images in the COCO dataset. Evaluations with discriminative ground truth for justification and human studies for discriminative image captioning reveal that our approach outperforms baseline generative and speaker-listener approaches for discrimination.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ALICE: Active Learning with Contrastive Natural Language Explanations (2020, methodology): “We adopt the random sampling method in (Vedantam et al., 2017), to make sure that the sampled species are challenging to classify.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “This framework has been applied to model human use of language [15], and to improve the performance of systems that generate [3, 30, 51, 12] and interpret [56, 29] referential language.”
Grounding Visual Explanations (2018, background): “Similarly, image descriptions are generated to make the target image distinguishable from a similar image in [28], and referential expressions are generated on objects in a discriminative way such that one can correctly localize the mentioned object from the generated expression in [17].”
Pragmatically Informative Text Generation (2019, background): “Pragmatic approaches in this category (Frank and Goodman, 2012; Andreas and Klein, 2016; Vedantam et al., 2017; Cohn-Gordon et al., 2018) derive pragmatic behavior by producing outputs that distinguish the input i from an alternate distractor input (or inputs).”
Explanation and Justification in Machine Learning : A Survey Or (2017, background): “[Vedantam et al. , 2017] produce captions that are locally discriminative, in the context of other images.”
Pragmatically Informative Image Captioning with Character-Level Inference (2018, methodology): “, 2017) and image captioning (Mao et al., 2016a; Vedantam et al., 2017), but these models do not fully confront the fact that the agents must reason about all possible utterances, which is intractable.”
Discriminability Objective for Training Descriptive Captions (2018, methodology): “The second additional set of results is for the model in [5], evaluated on captions provided by the authors.”
Continual Adaptation for Efficient Machine Communication (2019, methodology): “…reasoning allows listeners to infer richer intended meanings by considering counterfactual alternatives, and allows speakers to be appropriately informative, not merely truthful (Goodman and Frank, 2016; Andreas and Klein, 2016; Fried et al., 2018; Monroe et al., 2017; Vedantam et al., 2017).”
Translating Neuralese (2017, background): “, 2016), and approaches focused on generating explanations in the form of natural language (Hendricks et al., 2016; Vedantam et al., 2017).”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, background): “Recently, these approaches have also been adopted as a benchmark for discriminative and context-aware NLP [47,10,46,19,62,60,34,4].”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, background): “Image and video captioning [1, 6, 9, 11, 16, 18, 21, 35, 38, 39, 40] is the task of generating a textual description of an image, or similarly, ﬁnding the textual description that best matches the image.”
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts (2019, background): “…questions about entities in natural scenes (Antol et al., 2015; Hud-son and Manning, 2019), to perform discriminative and pragmatically-informative image captioning (Vedantam et al., 2017; Cohn-Gordon et al., 2018), and to discriminate between similar images in reference games (Su et al., 2017).”
Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models (2020, unknown): “…any image in the form of a sentence is also well known as image captioning and has been at the forefront of research for both computer vision and natural language processing (Vinyals et al., 2014; Karpathy & Feifei, 2014; Venugopalan et al., 2016; Selvaraju et al., 2019; Vedantam et al., 2017).”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “The differences are included in the description to make it more precise (Vedantam et al., 2017).”
Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning (2020, background): “The task we consider is essentially discriminative image captioning (Vedantam et al., 2017; Dai and Lin, 2017; Andreas and Klein, 2016).”