Auto-Encoding Scene Graphs for Image Captioning

By Xu Yang, Kaihua Tang, Hanwang Zhang, Jianfei Cai. 2018

801 citations recorded by the source corpus.

We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inference in discourse. For example, when we see the relation ``person on bike'', it is natural to replace ``on'' with ``ride'' and infer ``person riding bike on a road'' even the ``road'' is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models less likely to overfit to the dataset bias and focus on reasoning. Specifically, we use the scene graph --- a directed graph (G) where an object node is connected by adjective nodes and relationship nodes --- to represent the complex structural layout of both image (I) and sentence (S). In the textual domain, we use SGAE to learn a dictionary (D) that helps to reconstruct sentences in the S -> G -> D -> S pipeline, where D encodes the desired language prior; in the vision-language domain, we use the shared D to guide the encoder-decoder in the I -> G -> D -> S pipeline. Thanks to the scene graph representation and shared dictionary, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, \eg, our SGAE-based single-model achieves a new state-of-the-art 127.8 CIDEr-D on the Karpathy split, and a competitive 125.5 CIDEr-D (c40) on the official server even compared to other ensemble models. Code has been made available at: https://github.com/yangxuntu/SGAE.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, background): “describing the detailed semantics of images and captions, scene graphs have led to many state-of-the-art models in image captioning (Yang et al. 2019), image retrieval (Wu et al.”
Object-Centric Image Generation from Layouts (2020, methodology): “Scene-graphs and image matching Scene-graphs have been used traditionally as intermediate representations in image captioning [61,1], reconstruction [14] and retrieval [24], as well as in sentence to scene graph [49] and image to scene graph [33,39] prediction.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Besides, researchers have explored to leverage semantic attributes [26,53], scene graphs [50], and graph convolutional networks [52] for captioning.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, methodology): “Follow-up works further capture the relationships among object regions by using Graph Convolutional Networks (GCNs) (Yao et al. 2018), incorporating language inductive bias (Yang et al. 2019), or enforcing region grounding between image and text (Lu et al. 2018; Zhou et al. 2019).”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “Also, we compare with RFNet [13], which fuses encoded features from multiple CNN networks; SGAE [43], which introduces auto-encoding scene graphs into its model.”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “[46] used a multi-modal graph convolution network to modulate scene graphs into visual representations.”
Captioning Images Taken by People Who Are Blind (2020, methodology): “For instance, we observe better performance when UpDown [8] and AoANet [30] are trained from scratch on VizWiz-Captions rather than fine-tuned from models pretrained on MSCOCO-Captions, and vice versa for SGAE [56].”
Image Captioning: Transforming Objects into Words (2019, methodology): “Yang et al. in [27] similarly leveraged graph structures, extracting object image features into an image scene graph.”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “Scene graph generation (SGG) [62] — a visual detection task of objects and their relationships in an image — seems to have never fulfilled its promise: a comprehensive visual scene representation that supports graph reasoning for high-level tasks such as visual captioning [67, 65] and VQA [54, 15].”
Deconfounded Image Captioning: A Causal Retrospect (2020, methodology): “Then the captioners generate the words by weighted averaging the visual features at the selected positions at different time steps.”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, background): “Graph neural networks and scene graphs incorporated spatial as well as semantic relationships between objects [36, 72, 73].”
Visual Commonsense R-CNN (2020, neutral): “5 SGAE [71] 37.”
Say As You Wish: Fine-Grained Control of Image Caption Generation With Abstract Scene Graphs (2020, background): “A more structured representation over concepts, scene graph [18], is further explored [45, 46] in image captioning which can take advantage of detected objects and their relationships.”
Learning to Collocate Neural Modules for Image Captioning (2019, methodology): “When the dictionary preserving language bias is learned as in SGAE, even the query embeddings do not contain high-level semantic knowledge created by graph convolution network as SGAE, our CNM+SGAE also achieve better performances than SGAE.”
A Better Variant of Self-Critical Sequence Training (2020, methodology): “…2017; Dognin et al., 2019; Luo et al., 2018; Jiang et al., 2018; Liu et al., 2018b; Ling and Fi-dler, 2017; Liu et al., 2018a; Chen et al., 2019; Yang et al., 2019; Zhao et al., 2018), but also has been applied to many other tasks, like video cap-tioning(Li et al., 2018; Chen et al., 2018; Li…”
Set-Structured Latent Representations (2020, background): “As previously mentioned, many approaches have incorporated discrete structures to obtain better performance on a given task (Yang et al., 2019; Chen et al., 2019).”