Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

By Will Monroe, Robert D. Hawkins, Noah D. Goodman, Christopher Potts. 2017

145 citations recorded by the source corpus.

We present a model of pragmatic referring expression interpretation in a grounded communication task (identifying colors from descriptions) that draws upon predictions from two recurrent neural network classifiers, a speaker and a listener, unified by a recursive pragmatic reasoning framework. Experiments show that this combined pragmatic model interprets color descriptions more accurately than the classifiers from which it is built, and that much of this improvement results from combining the speaker and listener perspectives. We observe that pragmatic reasoning helps primarily in the hardest cases: when the model must distinguish very similar colors, or when few utterances adequately express the target color. Our findings make use of a newly-collected corpus of human utterances in color reference games, which exhibit a variety of pragmatic behaviors. We also show that the embedded speaker model reproduces many of these pragmatic behaviors.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Natural Language Processing, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Natural Language Processing

In Cognitive Science & Psychology

Citations in the corpus

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “While a number of methods [20, 21, 33] have been proposed to improve language understanding, common to all existing work is that the agent learns to understand each instruction from scratch or in isolation, without collectively leveraging prior visiongrounded domain knowledge.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “Following previous work on pragmatic language generation and interpretation [45, 3, 34, 16], we use a rescoring procedure: produce candidate route interpretations for a given instruction using the base follower model, and then rescore these routes using the base speaker model (Figure 2(c)).”
Continual Adaptation for Efficient Machine Communication (2019, background): “…reasoning allows listeners to infer richer intended meanings by considering counterfactual alternatives, and allows speakers to be appropriately informative, not merely truthful (Goodman and Frank, 2016; Andreas and Klein, 2016; Fried et al., 2018; Monroe et al., 2017; Vedantam et al., 2017).”
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts (2019, background): “…to explore in future work, is whether training models to jointly learn superlative, comparative, and positive GAs (similarly to how Pezzelle et al. (2018) did for quantities), or framing the task in a dialogue setting (as Monroe et al. (2017) did for colors) could lead to more compositional models.”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, methodology): “Recently, these approaches have also been adopted as a benchmark for discriminative and context-aware NLP [47,10,46,19,62,60,34,4].”
Pragmatically Informative Text Generation (2019, background): “Recent work has extended these models to more complex groundings, including images (Mao et al., 2015) and trajectories (Fried et al., 2018).”
Inferring Rewards from Language in Context (2022, methodology): “…in a single context, e.g., choosing the correct referent in signaling games (Golland et al., 2010; Frank and Goodman, 2012; Degen et al., 2013; Monroe et al., 2017; McDowell and Goodman, 2019), interpreting implicatures (Goodman and Stuhlmüller, 2013; Bergen et al., 2016), or generating…”