ReferItGame: Referring to Objects in Photographs of Natural Scenes

By Sahar Kazemzadeh, Vicente Ordonez, M. Matten, Tamara L. Berg. 2014

1,801 citations recorded by the source corpus.

In this paper we introduce a new game to crowd-source natural language referring expressions. By designing a two player game, we can both collect and verify referring expressions directly within the game. To date, the game has produced a dataset containing 130,525 expressions, referring to 96,654 distinct objects, in 19,894 photographs of natural scenes. This dataset is larger and more varied than previous REG datasets and allows us to study referring expressions in real-world scenes. We provide an in depth analysis of the resulting dataset. Based on our findings, we design a new optimization based model for generating referring expressions and perform experimental evaluations on 3 test sets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, background): “SCAN (Lee et al., 2018) and MAttNet (Yu et al., 2018) studied learning latent alignment between words and image regions for Image-Text Retrieval (Wang et al., 2016) and Referring Expression Comprehension (Kazemzadeh et al., 2014) tasks.”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “Various tasks such as visual question answering (Antol et al., 2015; Goyal et al., 2017), textual grounding (Kazemzadeh et al., 2014; Plummer et al., 2015), and visual reasoning (Suhr et al., 2019; Zellers et al., 2019) have been proposed and various models (Yang et al., 2016; Anderson et al.,…”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “…summer research project [1] Since this now famously ambitious summer project, steady progress has been made towards systems that can demonstrate their visual understanding by generating or responding to natural language in the context of images, videos, or even full 3D environments [2–8].”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “, image-text retrieval [60], visual question answering [6], visual grounding [28, 58], etc.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “To evaluate the performance of ERNIE-ViL, we conduct experiments on various vision-language tasks, (1) visual question answering (VQA 2.0 [11]), (2) visual commonsense reasoning (VCR [12]) (3) region-to-phrase grounding (RefCOCO+ [16]) (4) image-text/text-image retrieval (Flickr30K [17]).”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “[56] for a survey), with great progress on important problems such as visual question answering [3, 26] and image caption generation and retrieval [8, 88, 43, 68, 28], with offshoot tasks focusing specifically on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual”
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, background): “In recent years, a range of benchmarks for visual understanding have been proposed, including datasets for image captioning [7, 8, 23, 45], referring to objects [19], relational graph prediction [21], and visual Turing tests [12, 27].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “This dataset is designed for the task of referring expression comprehension [17].”
Unifying Vision-and-Language Tasks via Text Generation (2021, background): “, 2018), referring expression comprehension (Kazemzadeh et al., 2014; Yu et al., 2016; Mao et al., 2016), and image captioning (Chen et al.”
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “Lastly, we devise a ranking optimization module to further ﬁne-tune on the dense annotations ( § 3.3).”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Recent work has extended this to vision and language tasks [2, 45–50], leading to compelling results in Visual Question Answering [4], Commonsense Reasoning [51], Natural Language Visual Reasoning [52], Entailment [53], Image-Text Retrieval [54, 55], Referring Expressions [56], and Vision-Language N”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, methodology): “We adopt the RefCOCO+ (Kazemzadeh et al., 2014) dataset for evaluation, consisting of 141k expressions for 50k referred objects in 20k images in the COCO dataset (Lin et al.”
VQA: Visual Question Answering (2015, background): “Several recent papers have explored tasks at the intersection of vision and language that are easier to evaluate than image captioning, such as coreference resolution [22, 36] or generating referring expressions [20, 35] for a particular object in an image that”
Improved Baselines with Visual Instruction Tuning (2023, background): “Furthermore, we find further adding region-level VQA datasets (Visual Genome [24], RefCOCO [23, 39]) improves the model’s capability of localizing fine-grained visual details.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “However, although many tasks combining visual and linguistic reasoning have been motivated by their potential robotic applications [4, 14, 21, 30, 42], none of these tasks allow an agent to move or control the camera.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “, 2016), referring expression resolution (e.g., Mitchell et al., 2010; Kazemzadeh et al., 2014; Mao et al., 2016), visual entailment (Xie et al.”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, methodology): “For instance, a single epoch of VG (our largest dataset) corresponds to 19.8 epochs of RefCOCOg (our smallest).”