Pragmatically Informative Image Captioning with Character-Level Inference

By Reuben Cohn-Gordon, Noah D. Goodman, Christopher Potts. 2018

112 citations recorded by the source corpus.

We combine a neural image captioner with a Rational Speech Acts (RSA) model to make a system that is pragmatically informative: its objective is to produce captions that are not merely true but also distinguish their inputs from similar images. Previous attempts to combine RSA with neural image captioning require an inference which normalizes over the entire set of possible utterances. This poses a serious problem of efficiency, previously solved by sampling a small subset of possible utterances. We instead solve this problem by implementing a version of RSA which operates at the level of characters (“a”, “b”, “c”, ...) during the unrolling of the caption. We find that the utterance-level effect of referential captions can be obtained with only character-level decisions. Finally, we introduce an automatic method for testing the performance of pragmatic speaker models, and show that our model outperforms a non-pragmatic baseline as well as a word-level RSA captioner.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “GeDi most closely relates to distractor based pragmatics (Andreas and Klein, 2016; Cohn-Gordon et al., 2018; Shen et al., 2019), where a single model processes a true input and a distractor input, and uses Bayes rule to produce text that fits the true input but not the distractor input.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “This framework has been applied to model human use of language [15], and to improve the performance of systems that generate [3, 30, 51, 12] and interpret [56, 29] referential language.”
FUDGE: Controlled Text Generation With Future Discriminators (2021, background): “, G’s output logits), and operates directly on these logits (Ghazvininejad et al., 2017; Holtzman et al., 2018; Cohn-Gordon et al., 2018; Shen et al., 2019).”
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts (2019, background): “, 2015; Hudson and Manning, 2019), to perform discriminative and pragmatically-informative image captioning (Vedantam et al., 2017; Cohn-Gordon et al., 2018), and to discriminate between similar im-”
Pragmatically Informative Text Generation (2019, methodology): “Pragmatic approaches in this category (Frank and Goodman, 2012; Andreas and Klein, 2016; Vedantam et al., 2017; Cohn-Gordon et al., 2018) derive pragmatic behavior by producing outputs that distinguish the input i from an alternate distractor input (or inputs).”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, background): “Recently, these approaches have also been adopted as a benchmark for discriminative and context-aware NLP [47,10,46,19,62,60,34,4].”
Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning (2020, background): “This reranker model is closely related to pragmatic speakers in Rational Speech Act (RSA) framework (Andreas and Klein, 2016; Monroe and Potts, 2015; Cohn-Gordon et al., 2018; Fried et al., 2018).”
Continual Adaptation for Efficient Machine Communication (2019, methodology): “This inversion is based on models of one-shot pragmatic inference in reference games (Goodman and Frank, 2016; Andreas and Klein, 2016; Vedantam et al., 2017; Cohn-Gordon et al., 2018).”