Obj2Text: Generating Visually Descriptive Language from Object Layouts

By Xuwang Yin, Vicente Ordonez. 2017

56 citations recorded by the source corpus.

Generating captions for images is a task that has recently received considerable attention. Another type of visual inputs are abstract scenes or object layouts where the only information provided is a set of objects and their locations. This type of imagery is commonly found in many applications in computer graphics, virtual reality, and storyboarding. We explore in this paper OBJ2TEXT, a sequence-to-sequence model that encodes a set of objects and their locations as an input sequence using an LSTM network, and decodes this representation using an LSTM language model. We show in our paper that this model despite using a sequence encoder can effectively represent complex spatial object-object relationships and produce descriptions that are globally coherent and semantically relevant. We test our approach for the task of describing object layouts in the MS-COCO dataset by producing sentences given only object annotations. We additionally show that our model combined with a state-of-the-art object detector can improve the accuracy of an image captioning model.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, methodology): “For panoptic and referring segmentation, we use COCO2017 [48] with segmentation annotations and exclude the validation sets of Ref-COCOg UMD [92] and COCO Karpathy [89].”
TVQA: Localized, Compositional Video Question Answering (2018, background): “Visual Concept Features: Recent work (Yin and Ordonez, 2017) found that using detected object”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Predicting such graph representations has been shown to improve natural language based image tasks [17, 38, 46] and has the potential to signiﬁcantly expand the scope of applications for computer vision systems.”
Text2Scene: Generating Compositional Scenes From Textual Descriptions (2018, methodology): “We generate captions from the semantic layouts using [36] and compare them back to the original captions used to generate the scenes.”
BERT Representations for Video Question Answering (2020, unknown): “Recent work [45, 21] has found that using detected object labels as input has comparable or better performance to using CNN features directly in image captioning and video-QA tasks.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Initially, (Yin & Ordonez, 2017) generated layout from images, while (Wang et al., 2018b) count image objects to produce multiple captions for a given image.”
On Modality Bias in the TVQA Dataset (2020, unknown): “Yin and Ordonez [49] found that simply using detected labels instead of image features gives comparable performance on image captioning tasks.”
Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering (2019, methodology): “Visual Concept Feature: Inspired by recent works [15], [39] that use detected object labels as visual inputs instead of using CNN features directly, we also extracted detected labels which are referred to as visual concept feature [15].”
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, unknown): “For example, detected object labels have been employed for image captioning [53, 4] and also video QA [26].”