Scene Graph Parsing as Dependency Parsing

By Yu-Siang Wang, Chenxi Liu, Xiaohui Zeng, Alan L. Yuille. 2018

59 citations recorded by the source corpus.

In this paper, we study the problem of parsing structured knowledge graphs from textual descriptions. In particular, we consider the scene graph representation that considers objects together with their attributes and relations: this representation has been proved useful across a variety of vision and language applications. We begin by introducing an alternative but equivalent edge-centric view of scene graphs that connect to dependency parses. Together with a careful redesign of label and action space, we combine the two-stage pipeline used in prior work (generic dependency parsing followed by simple post-processing) into one, enabling end-to-end training. The scene graphs generated by our learned neural dependency parser achieve an F-score similarity of 49.67% to ground truth graphs on our evaluation set, surpassing best previous approaches by 5%. We further demonstrate the effectiveness of our learned parser on image retrieval applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanwang Zhang
2. Xu Yang
3. Jianfei Cai
4. Haifeng Wang
5. Kaihua Tang
6. Hua Wu
7. Jiuxiang Gu
8. Hao Tian

In Computer Vision and Pattern Recognition

1. Jianfei Cai
2. Hanwang Zhang
3. Xu Yang
4. Kaihua Tang
5. Jiuxiang Gu
6. Haifeng Wang
7. Hua Wu
8. Zhe L. Lin

In Natural Language Processing

1. Hanwang Zhang
2. Xu Yang
3. Jianfei Cai
4. Haifeng Wang
5. Hua Wu
6. Hao Tian
7. Yu Sun
8. Kaihua Tang

Citations in the corpus

• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Various methods have been proposed to parse scene graphs from images [26] [27] and texts [28] [29] [30].”
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, methodology): “For phrase-level and word-level rewriting, we first parse the image caption into a scene graph (Wang et al., 2018), then randomly replacing the object, attribute or relation nodes of the scene graph with a different object, attribute or relation from the corresponding vocabularies.”
• Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “In particular, we use scene graphs [13, 47] to bridge the gap between the two worlds.”
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, unknown): “In the vision community, we have also seen attempts to construct the so-called scene graph [31, 38] that can describe a visual scene in a similar way to a knowledge graph [44, 45, 65, 67, 69, 77].”
• Scene Graph Generation With External Knowledge and Image Reconstruction (2019, methodology): “Like other approaches [26, 25, 37], during training we want the generated scene graph close to the ground-truth scene graph by optimizing the scene graph generation process with object detection loss and relationship classification loss”
• Know More Say Less: Image Captioning Based on Scene Graphs (2019, background): “[58] introduce a new perspective and solution for the task of generating scene graphs from textual descriptions.”