A Joint Speaker-Listener-Reinforcer Model for Referring Expressions

By Licheng Yu, Hao Hao Tan, Mohit Bansal, Tamara L. Berg. 2016

294 citations recorded by the source corpus.

Referring expressions are natural language constructions used to identify particular objects within a scene. In this paper, we propose a unified framework for the tasks of referring expression comprehension and generation. Our model is composed of three modules: speaker, listener, and reinforcer. The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions. The listener-speaker modules are trained jointly in an end-to-end learning framework, allowing the modules to be aware of one another during learning while also benefiting from the discriminative reinforcer’s feedback. We demonstrate that this unified framework and training achieves state-of-the-art results for both comprehension and generation on three referring expression datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Explain With Complemental Examples (2018, neutral): “,[28]) in that they utilize auxiliary models.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “hances of successful communicative outcomes. This framework has been applied to model human use of language [15], and to improve the performance of systems that generate [3, 31, 53, 12] and interpret [58, 30, 52] referential language. Similar modeling tools have recently been applied to generation a”
Translating Neuralese (2017, background): “Communication strategies for reference games were previously explored by Vogel et al. (2013a), Andreas and Klein (2016) and Kazemzadeh et al. (2014), and reference games speciﬁcally featuring end-to-end communication protocols by Yu et al. (2016).”
Visual Grounding with Transformers (2021, methodology): “Method Backbone RefCOCO RefCOCO+ RefCOCOg Flickr30K val testA testB val testA testB val-g val-u test-u test proposal-based: SLR [31] ResNet101 69.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “High-level concept words (Yu et al., 2017b) are detected such that they can be integrated with any video and language models addressing fill-in-the blank and multiple-choice test.”
VQD: Visual Query Detection In Natural Scenes (2019, unknown): “To aid in this endeavor, we also include comparison results directly from the SLR models (Yu et al., 2017) for RER, which is a recent system for that task.”
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, methodology): “RefCOCO RefCOCO+ model testA testB model testA testB Spe+Lis+Rl [89] 73.”
Referring Image Segmentation via Recurrent Refinement Networks (2018, methodology): “The method in [34] jointly learns speaker and listener modules via discriminative reinforcer’s feedback.”
Ferret: Refer and Ground Anything Anywhere at Any Granularity (2023, background): “, 2019), whereas grounding necessitates the model to localize the region in accordance with the given semantic description (Luo & Shakhnarovich, 2017; Nagaraja et al., 2016; Yu et al., 2017; Kamath et al., 2021).”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, methodology): “Referring Expression Comprehension (REC) [11, 12, 19, 21, 44, 45, 48, 42, 37] and Referring Expression Segmentation (RES) [32, 16, 40, 25, 34] are two emerging tasks, which involves identifying the target visual instances according to a given linguistic expression.”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “The first one is multi-stage modeling [6, 49, 20, 19, 69, 67], which typically regards REC as a metric learning problem, i.”
Learning to Write with Cooperative Discriminators (2018, methodology): “Yu et al. (2017c) proposed a model where the speaker consists of a convolutional encoder and an LSTM decoder, trained with a ranking loss on negative samples in addition to optimizing log-likelihood.”
Discriminability Objective for Training Descriptive Captions (2018, methodology): “[34, 35] learned to generate more discriminative referring expressions guided by a referring expression comprehension model.”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “03 7 Speaker+Listener+Reinforcer [32] vgg16 79.”