Scene Graph Generation from Objects, Phrases and Region Captions

By Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, Xiaogang Wang. 2017

551 citations recorded by the source corpus.

Object detection, scene graph generation and region captioning, which are three scene understanding tasks at different semantic levels, are tied together: scene graphs are generated on top of objects detected in an image with their pairwise relationship predicted, while region captioning gives a language description of the objects, their attributes, relations and other context information. In this work, to leverage the mutual connections across semantic levels, we propose a novel neural network model, termed as Multi-level Scene Description Network (denoted as MSDN), to solve the three vision tasks jointly in an end-to-end manner. Object, phrase, and caption regions are first aligned with a dynamic graph based on their spatial and semantic connections. Then a feature refining structure is used to pass messages across the three levels of semantic tasks through the graph. We benchmark the learned model on three tasks, and show the joint learning across three tasks with our proposed method can bring mutual improvements over previous models. Particularly, on the scene graph generation task, our proposed method outperforms the stateof- art method with more than 3% margin. Code has been made publicly available.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, methodology): “It is common to build scene graphs for static images [14,32,34,61,62,64].”
Exploring Visual Relationship for Image Captioning (2018, methodology): “In the literature, it is well recognized that reasoning such visual relationships is crucial to a richer semantic understanding [19,23] of the visual world.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, methodology): “ough conditioning and avoids uninformative edge predictions until the end. Others have explored creating richer models between image regions, introducing new convolutional features and new objectives [31,57,25, 27]. Our work is complementary and instead focuses on the role of context. See the supple”
Visual Question Generation as Dual Task of Visual Question Answering (2017, methodology): “Inspired by the previous works on visual relationships [17, 18], the image can be viewed as a visual relationship 〈tie-aroundneck〉.”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Previous research has proposed plenty of methods for predicting scene graphs on static images [48, 52, 77, 78, 85, 86].”
Graph R-CNN for Scene Graph Generation (2018, methodology): “To be consistent with previous work [18,40], we also adopt the two-stage training – we ﬁrst train the object detector and then train the whole model jointly until convergence.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “Xu et al. (2017) and Li et al. (2017) develop algorithms for 2D scene graph generation.”
Deep Learning for Generic Object Detection: A Survey (2018, background): “o efﬁciently and effectively incorporate contextual information remains to be explored, ideally guided by how human use the context in recognizing objects. The recent progress on building scene graph [157] may be another way of building structured understanding of objects in the scene. The full segm”
LinkNet: Relational Embedding for Scene Graph (2018, unknown): “Compared to previous models [33, 18] that have been proposed to focus on message passing between nodes and edges, our model explicitly reasons about the relations within nodes and edges and predicts graph elements in multiple steps [35], such that features of a previous stage provides rich context…”
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships (2020, methodology): “Inspired by image captioning methods which seek to use scene graph and object relationships [10, 22, 42] , we propose to fully leverage the spatial relationship between close-by objects instead of individually focusing on information-constrained occluded objects.”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, unknown): “We conduct comprehensive empirical studies of GNs on the Visual Genome dataset [14, 38], which provides human annotated scene graphs. Recent algorithms on automatic scene graph generation from images [44,45,65,67,69,77] have also focused on Visual Genome, allowing us to evaluate these algorithms usi”
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation (2018, methodology): “All the experiments are performed on VG-MSDN [4] as it is larger than VRD [1] to eliminate overfitting and contains more predicate categories than VG-DR-Net [6].”
Embodied Semantic Scene Graph Generation (2021, methodology): “For example, the Visual Genome dataset [3], which contains annotations of objects and their relationships in images, has been extensively utilized in various scene graph generation tasks [4][5][6].”
Question-Guided Hybrid Convolution for Visual Question Answering (2018, methodology): “Recently, CNN and RNN are jointly trained for learning feature representations for multi-modal tasks, including image captioning [3,4], text-to-image retrieval [5,33], and Visual Question Answering (VQA) [6,11,12,38].”
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera (2019, background): “Examples are on scene graph generation [30, 46], image cap-tioning/description [26, 3, 23], image retrieval [24] and visual question-answering [17, 51].”
Image Scene Graph Generation (SGG) Benchmark (2021, methodology): “We have integrated ﬁve widely used scene graph generation algorithms into our framework, including iterated message passing (IMP) [18], multi-level scene detection networks (MSDN) [12], graph R-CNN (GRCNN) [19], neural motif (NM) [21] and RelDN [22].”