Scene Graph Generation With External Knowledge and Image Reconstruction

By Jiuxiang Gu, Handong Zhao, Zhe L. Lin, Sheng Li, Jianfei Cai, Mingyang Ling. 2019

321 citations recorded by the source corpus.

Scene graph generation has received growing attention with the advancements in image understanding tasks such as object detection, attributes and relationship prediction, etc. However, existing datasets are biased in terms of object and relationship labels, or often come with noisy and missing annotations, which makes the development of a reliable scene graph prediction model very challenging. In this paper, we propose a novel scene graph generation algorithm with external knowledge and image reconstruction loss to overcome these dataset issues. In particular, we extract commonsense knowledge from the external knowledge base to refine object and phrase features for improving generalizability in scene graph generation. To address the bias of noisy object annotations, we introduce an auxiliary image reconstruction path to regularize the scene graph generation network. Extensive experiments show that our framework can generate better scene graphs, achieving the state-of-the-art performance on two benchmark datasets: Visual Relationship Detection and Visual Genome datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, background): “2016), reconstruction (Gu et al. 2019) and retrieval (Johnson et al.”
Unbiased Scene Graph Generation From Biased Training (2020, critical): “Most of the existing methods [62, 60, 7, 26, 68, 53, 64, 10, 43, 59] struggle for better feature extraction networks.”
Visual Semantic Reasoning for Image-Text Matching (2019, background): “A large amount of research [5, 9, 23] has been done to bridge these two modalities.”
Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “When it comes to utilizing commonsense, existing methods integrate it within the inference pipeline, either by retrieving a set of relevant facts from a knowledge base and feeding as additional features to the model [18,7,22], or by employing a graph-based message propagation process to embed the st”
Learning to Collocate Neural Modules for Image Captioning (2019, background): “Indeed, image captioning is not the only model that can easily exploit the dataset bias to captioning even without looking at the image [44], almost all existing models for visual reasoning tasks such as visual Q&A [23, 46, 48] have been spotted mode collapse to certain dataset idiosyncrasies and fa”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “Prior knowledge has been proven to significantly improve the quality of SGG. Existing methods use either an external curated knowledge base, such as ConceptNet [123], [126], [127], [128], or the statistical information found in the annotation corpus to obtain commonsense data.”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, background): “Other works focus on leveraging external information [18] or explicit scene graph [5] to extract features from input images.”
Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks (2019, methodology): “Furthermore, self-supervised auxiliary tasks have been widely applied in the fields of computer vision [45, 12, 27], natural language processing [9, 19] and meta learning [40, 20].”