Exploring Visual Relationship for Image Captioning

By Ting Yao, Yingwei Pan, Yehao Li, Tao Mei. 2018

923 citations recorded by the source corpus.

It is always well believed that modeling relationships between objects would be helpful for representing and eventually describing an image. Nevertheless, there has not been evidence in support of the idea on image description generation. In this paper, we introduce a new design to explore the connections between objects for image captioning under the umbrella of attention-based encoder-decoder framework. Specifically, we present Graph Convolutional Networks plus Long Short-Term Memory (dubbed as GCN-LSTM) architecture that novelly integrates both semantic and spatial object relationships into image encoder. Technically, we build graphs over the detected objects in an image based on their spatial and semantic connections. The representations of each region proposed on objects are then refined by leveraging graph structure through GCN. With the learnt region-level features, our GCN-LSTM capitalizes on LSTM-based captioning framework with attention mechanism for sentence generation. Extensive experiments are conducted on COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GCN-LSTM increases CIDEr-D performance from 120.1% to 128.7% on COCO testing set.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “[59], including on, cover, overlap, eight-way relative orientation, and unrelated.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Besides, researchers have explored to leverage semantic attributes [26,53], scene graphs [50], and graph convolutional networks [52] for captioning.”
ClipCap: CLIP Prefix for Image Captioning (2021, background): “Many approaches have been proposed for image captioning [4, 9, 13, 19, 34, 35, 42, 44, 47].”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, methodology): “Follow-up works further capture the relationships among object regions by using Graph Convolutional Networks (GCNs) (Yao et al. 2018), incorporating language inductive bias (Yang et al.”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “[12] have explored the incorporation of relationships between image regions, both from a semantic point of view and using geometric features such as the position and the spatial extent of the region.”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, background): “[42, 26] modeled intra-modality object-to-object relations for improving image captioning performance.”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “Moreover, our small batch size and fewer training epochs still lead to higher performances than GCN-LSTM, whose batch size and training epochs are much larger.”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “[48] have proposed to use a graph convolution neural network in the image encoding phase to integrate semantic and spatial relationships between objects.”
XGPT: Cross-modal Generative Pre-Training for Image Captioning (2020, neutral): “CO C B@4 M S C B@4 M S Approaches that do not use any pre-trained model BUTD [23] 56.6 27.3 21.7 16.0 113.5 36.2 27.0 20.3 NBT (with BBox) [31] 57.5 27.1 21.7 15.6 107.2 34.7 27.1 20.1 GCN-LSTM (spa) [32] - - - - 115.6 36.5 27.8 20.9 GCN-LSTM (sem) [32] - - - - 116.3 36.8 27.9 20.9 GVD [33] 62.3 27.”
Image Captioning: Transforming Objects into Words (2019, methodology): “As a way of adding global context, Yao et al. in [29] introduced two Graph Convolutional Networks: a semantic relationship graph, and a spatial relationship graph that classiﬁes the relationship between two boxes into 11 classes, such as “inside”, “cover”, or “overlap”.”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “Scene graph generation (SGG) [62] — a visual detection task of objects and their relationships in an image — seems to have never fulfilled its promise: a comprehensive visual scene representation that supports graph reasoning for high-level tasks such as visual captioning [67, 65] and VQA [54, 15].”
Visual Semantic Reasoning for Image-Text Matching (2019, methodology): “[39] needs to train the relationship detection model on Visual Genome).”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, methodology): “We follow [49] and use a graph convolutional layer that utilizes two different weight matrices to capture top-down and bottom-up information flow separately.”
Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, methodology): “Some researches in the ﬁeld of Visual Question Answering, Image Captioning even Action Recognition demonstrate that the relationship between objects is vital, which also plays an important role in generating a more detailed and diverse description for a video.”
Two Causal Principles for Improving Visual Dialog (2019, neutral): “Visual Dialog [9, 10] is more interactive and challenging than most of the vision-language tasks, e.g ., image captioning [45, 44, 4] and VQA [5, 38, 37, 36].”
The Supplementary Material: Counterfactual Contrastive Learning for Weakly-Supervised Vision-Language Grounding (2020, methodology): “We ﬁrst deﬁne a directed spatial graph with multi-type edges as in [15, 14, 13].”
Aligning Visual Regions and Textual Concepts: Learning Fine-Grained Image Representations for Image Captioning (2019, methodology): “A most recent advance (Yao et al., 2018) built graphs over the RCNN-detected visual regions, whose relationships are modeled as directed edges in a scene-graph, which is further encoded via a Graph Convolutional Network (GCN).”
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration (2021, methodology): “Inspired by [14, 20, 43], we calculate the similarity between each paired regions by their Intersection over Union (IoU) score.”