Reasoning about Pragmatics with Neural Listeners and Speakers

By Jacob Andreas, Daniel Klein. 2016

190 citations recorded by the source corpus.

We present a model for pragmatically describing scenes, in which contrastive behavior results from a combination of inference-driven pragmatics and learned semantics. Like previous learned approaches to language generation, our model uses a simple feature-driven architecture (here a pair of neural "listener" and "speaker" models) to ground language in the world. Like inference-driven approaches to pragmatics, our model actively reasons about listener behavior when selecting utterances. For training, our approach requires only ordinary captions, annotated _without_ demonstration of the pragmatic behavior the model ultimately exhibits. In human evaluations on a referring expression game, our approach succeeds 81% of the time, compared to a 69% success rate using existing techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, methodology): “Recently, Suhr et al. (2019a) introduced a testbed for evaluating language understanding in the service of a shared goal, and Andreas and Klein (2016) use a simpler visual paradigm for studying pragmatics.”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “GeDi most closely relates to distractor based pragmatics (Andreas and Klein, 2016; Cohn-Gordon et al., 2018; Shen et al., 2019), where a single model processes a true input and a distractor input, and uses Bayes rule to produce text that fits the true input but not the distractor input.”
Learning to Write with Cooperative Discriminators (2018, background): “Andreas and Klein (2016) proposed a neural model where candidate descriptions are sampled from a genera-tively trained speaker , which are then re-ranked by interpolating the score with that of the listener , a discriminator that predicts a distribution over choices given the speaker’s description.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “Following previous work on pragmatic language generation and interpretation [45, 3, 34, 16], we use a rescoring procedure: produce candidate route interpretations for a given instruction using the base follower model, and then rescore these routes using the base speaker model (Figure 2(c)).”
Grounding Visual Explanations (2018, background): “Our work is also related to the recent work of pragmatics-oriented language generation [1] where a describer produces a set of sentences, then a choice ranker chooses which sentence best fulfills a specific goal, e.”
Adversarial Language Games for Advanced Natural Language Intelligence (2019, unknown): “There are also some works (Vogel et al., 2013; Smith et al., 2013; Monroe and Potts, 2015; Hawkins et al., 2015; Andreas and Klein, 2016; Khani et al., 2018) study the pragmatic reasoning ability of NLP”
Mapping natural language commands to web elements (2018, background): “At the same time, our task can be viewed as a reference game (Golland et al., 2010; Smith et al., 2013; Andreas and Klein, 2016), where the system has to select an object given a natural language reference.”
Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images (2017, unknown): “Using policy gradient algorithms with the reinforce rule/trick [43] allows estimation of gradients through discrete samples [16, 2, 46, 24].”
Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding (2017, background): “The RSA recursion is then defined in terms of these base agents: the pragmatic speaker produces utterances based on a literal RNN listener (Andreas and Klein, 2016), and the pragmatic listener interprets utterances based on the pragmatic speaker’s behavior.”
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions (2016, methodology): “Most related work to us is [1], which uses a speaker model to generate expressions for abstract images then uses an offline listener to rerank the speaker’s output.”
Executing Instructions in Situated Collaborative Interactions (2019, background): “CEREALBAR also provides opportunities to study pragmatic reasoning for language understanding (Andreas and Klein, 2016; Fried et al., 2018; Liang et al., 2019).”
Interactive Learning from Activity Description (2021, methodology): “Second, we apply prag-4 matic inference (Andreas & Klein, 2016; Fried et al., 2018a), leveraging the fact that the teacher has access to the environment’s simulator and can simulate executions of descriptions.”
Pragmatic Language Interpretation as Probabilistic Inference. (2016, unknown): “These developments may go together with new algorithms for learning aspects of the underlying semantics, which will open up new applications for the RSA approach in computational linguistics and artificial intelligence [31, 76, 57, 1, 61].”
Continual Adaptation for Efficient Machine Communication (2019, methodology): “As in previous computational approaches to pragmatics (e.g. Goodman and Frank, 2016; Andreas and Klein, 2016), we formulate this problem as an inference about another agent.”
Pragmatically Informative Text Generation (2019, methodology): “Our work builds on a line of learned Rational Speech Acts (RSA) models (Monroe and Potts, 2015; Andreas and Klein, 2016), in which generated strings are selected to optimize the behavInput meaning representation (i): NAME[FITZBILLIES], EATTYPE[COFFEE SHOP], FOOD[ENGLISH], PRICERANGE[CHEAP], CUSTOMER”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, background): “Recently, these approaches have also been adopted as a benchmark for discriminative and context-aware NLP [47,10,46,19,62,60,34,4].”
Pragmatically Informative Image Captioning with Character-Level Inference (2018, background): “This core idea underlies much work in referring expression generation (Dale and Reiter, 1995; Monroe and Potts, 2015; Andreas and Klein, 2016; Monroe et al., 2017) and image captioning (Mao et al.”
Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning (2020, background): “This reranker model is closely related to pragmatic speakers in Rational Speech Act (RSA) framework (Andreas and Klein, 2016; Monroe and Potts, 2015; Cohn-Gordon et al., 2018; Fried et al., 2018).”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “Finally, it may be possible to extend our approach to other linguistic tasks such as dependency parsing (Christie et al., 2016b), coreference resolution (Kottur et al., 2018), and learning pragmatics beyond semantics (Andreas and Klein, 2016).”