Generation and Comprehension of Unambiguous Object Descriptions
By Junhua Mao, Jonathan Huang, Alexander Toshev, Oana-Maria Camburu, Alan L. Yuille, Kevin P. Murphy. 2015
1,750 citations recorded by the source corpus.
We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MSCOCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/ mjhucla/Google_Refexp_toolbox.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Luke S. Zettlemoyer
- Devi Parikh
- Percy Liang
- Zhe Gan
- Christopher D. Manning
In Natural Language Generation
- Marcus Rohrbach
- Trevor Darrell
- Lisa Anne Hendricks
- Kate Saenko
- Emiel Krahmer
- Peter Anderson
- Jacob Andreas
- Mark Johnson
Citations in the corpus
- Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “, 2018b) in a wide range of discriminative (Goyal et al., 2019; Hudson & Manning, 2019; Lei et al., 2018; Mao et al., 2016; Xu et al., 2016; Zhou et al., 2018) and generative tasks (Chen et al.”
- Improved Baselines with Visual Instruction Tuning (2023, background): “Furthermore, we find further adding region-level VQA datasets (Visual Genome [24], RefCOCO [23, 39]) improves the model’s capability of localizing fine-grained visual details.”
- Generalized Decoding for Pixel, Image, and Language (2022, background): “Since the emergence of vision transformers, works like LAVT [86] enhance the cross-modal interactions from the very beginning, which led to SoTA on RefCOCO [92], RefCOCO+ [92] and G-Ref [56,60].”
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “However, although many tasks combining visual and linguistic reasoning have been motivated by their potential robotic applications [4, 14, 21, 30, 42], none of these tasks allow an agent to move or control the camera.”
- From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “An orthogonal line of work has been on referring expressions: asking to what image region a natural language sentence refers to [60, 52, 65, 87, 88, 59, 36, 33].”
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “instance in an image. It augments the IAPR-TC dataset (Grubinger et al. 2006) of 20k photographs with 130k isolated entity descriptions for 97k objects from 238 categories. The Google Refexp dataset (Mao et al. 2016) is built on top of MSCOCO and contains a little under 27k images with 105k descript”
- 12-in-1: Multi-Task Vision and Language Representation Learning (2019, methodology): “Rel(vi, Q) = Wrhvi (3) Note thatQmay be either a phrase, question or dialog based on different tasks (RefCOCO+/g, Visual7W, GuessWhat).”
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, neutral): “Visual information retrieval with natural language queries has long been tackled via joint visuallanguage embedding models [12,19,24,25,26,27,28].”
- Generating Visual Explanations (2016, background): “[28] proposes an alternative training scheme for generating unambiguous region descriptions which maximizes the probability of a specific region description while minimizing the probability of other region descriptions.”
- Localizing Moments in Video with Natural Language (2017, methodology): “Natural language object retrieval [14, 22] can be seen as an analogous task to ours, where natural language phrases are localized spatially in images, rather than temporally in videos.”
- Grounding of Textual Phrases in Images by Reconstruction (2015, neutral): “un phrases present in textual descriptions. [1] report the localization performance of their proposed CCA embedding [14] approach. The Spatial Context Recurrent ConvNet (SCRC) [6] and the approach of [15] use a caption generation framework to score the phrase on the set of proposal boxes, to select ”
- Natural Language Object Retrieval (2015, background): “Concurrent with our work, [23] also proposes a recurrent network model to localize objects from given descriptions.”
- Modeling Context in Referring Expressions (2016, methodology): “The most relevant work to ours is Mao et al [26] which introduced the first deep learning approach to REG.”
- Neural Baby Talk (2018, background): “Another related line of work is on resolving referring expressions [21] (or description-based object retrieval [36, 17, 18, 39] – given a description of an object in the image, identify which object is being referred to) or referring expression generation [21, 29, 31, 50] (given an object in the ima”
- MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, methodology): “Both RefCOCO and RefCOCO+ were collected in a timed game setting whereas RefCOCOg was not.”
- Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “This framework has been applied to model human use of language [15], and to improve the performance of systems that generate [3, 30, 51, 12] and interpret [56, 29] referential language.”