Modeling Context in Referring Expressions

By Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, Tamara L. Berg. 2016

1,767 citations recorded by the source corpus.

Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images. In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets - RefCOCO, RefCOCO+, and RefCOCOg (Datasets and toolbox can be downloaded from https://github.com/lichengunc/refer), shows the advantages of our methods for both referring expression generation and comprehension.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “For Referring Expression (RE) Comprehension, we evaluate on RefCOCO, RefCOCO+, and RefCOCOg datasets [17], where given a text description, the model selects the described region from a set of image region proposals.”
Unifying Vision-and-Language Tasks via Text Generation (2021, background): “, 2018), referring expression comprehension (Kazemzadeh et al., 2014; Yu et al., 2016; Mao et al., 2016), and image captioning (Chen et al.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “For panoptic and referring segmentation, we use COCO2017 [46] with segmentation annotations and exclude the validation sets of Ref-COCOg UMD [86] and COCO Karpathy [84].”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “An orthogonal line of work has been on referring expressions: asking to what image region a natural language sentence refers to [60, 52, 65, 87, 88, 59, 36, 33].”
TVQA: Localized, Compositional Video Question Answering (2018, background): “, 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “While most previous work [31, 32, 18, 19] evaluates CNN features for each region proposal/candidate object, we use Faster R-CNN [20] as the backbone net for a faster and more principled implementation.”
Neural Baby Talk (2018, background): “Another related line of work is on resolving referring expressions [21] (or description-based object retrieval [36, 17, 18, 39] – given a description of an object in the image, identify which object is being referred to) or referring expression generation [21, 29, 31, 50] (given an object in the ima”
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, methodology): “There are three established datasets for this task called RefCOCO, RefCOCO+ [70] and Ref-COCOg [36].”
GuessWhat?! Visual Object Discovery through Multi-modal Dialogue (2016, methodology): “To do so, we follow the approach of [16, 46] and extract an 8-dimensional vector of the location of the bounding box: xspatial = [xmin, ymin, xmax, ymax, xcenter, ycenter, wbox, hbox], where wbox and hbox denote the width and height of the bounding box, respectively.”
Modeling Relationships in Referential Expressions with Compositional Modular Networks (2016, methodology): “The problem of grounding referential expressions can be naturally formulated as a retrieval problem over image regions [19, 10, 24, 7, 32, 20].”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, background): “Visual grounding tasks like referring expression understanding [10], [11] and phrase localization [12] find image regions indicated by questions, phrases, or sentences.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “…[73] for visual entailment, and multimodal generation including MSCOCO Image Caption [74] for image captioning, RefCOCO / RefCOCO+ / RefCOCOg [75, 76] for referring expression comprehension as this To evaluate the capability of visual grounding, we conduct experiments on RefCOCO, RefCOCO+,…”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “, captioning an image or a video with natural language [9, 10, 39, 41, 47, 48, 42] or localizing desired objects within an image given a natural language description [31, 18, 49].”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, background): “Earlier works [16, 13, 45, 44, 42, 32] focused on identifying the referred object in an image.”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “When ﬁnetuned on downstream tasks, these pre-trained models have achieved new state of the art on image-text retrieval [18], visual question answering [4,10], referring expression comprehension [23,39], visual reasoning [13,29,40], and visual entailment [38].”
Modeling Context Between Objects for Referring Expression Understanding (2016, neutral): “We perform experiments on the Google RefExp dataset [4] and the UNC RefExp dataset [10].”
Guide Me: Interacting with Deep Networks (2018, background): “Other examples include generation of referring expressions [36, 44, 47, 62], segmentation or object retrieval from referring expressions [19, 29, 28], image captioning [16, 21, 33, 34, 48, 56] and visual dialog [18].”