Scene Graph Generation by Iterative Message Passing

By Danfei Xu, Yuke Zhu, Christopher Bongsoo Choy, Li Fei-Fei. 2017

1,425 citations recorded by the source corpus.

Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel end-to-end model that generates such structured scene representation from an input image. Our key insight is that the graph generation problem can be formulated as message passing between the primal node graph and its dual edge graph. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods on the Visual Genome dataset as well as support relation inference in NYU Depth V2 dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Various methods have been proposed to parse scene graphs from images [26] [27] and texts [28] [29] [30].”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “Since sets are closely related to graphs, there are also natural extensions to molecule generation (Simonovsky & Komodakis, 2018) and scene graph prediction (Krishna et al., 2017; Xu et al., 2017).”
Grounded Language-Image Pre-training (2021, background): “However, to gain fine-grained understanding of images, as required by many tasks, such as object detection [33, 46], segmentation [7,37], human pose estimation [51,58], scene understanding [15, 27, 59], action recognition [19], visionlanguage understanding [8,29–32,38,50,52,65,67], objectlevel visua”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, methodology): “d embedding distances to get preliminary annotations, which are then followed by manual curation. This results in a class hierarchy 3We use the cleaner 1.4 version of the dataset, following Xu et al. [39] 4We expand the original Visual Genome dataset with 5k new scene graphs collected through crowds”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “Scene graph based relationship modeling, using a structured representation for describing object relationships and their attributes [35, 43, 44, 74] has been extensively studied.”
Continuous Scene Representations for Embodied AI (2022, background): “Beyond the inherent limitations of modeling relationships as discrete symbols, scene graphs used in the literature are typically static [28, 34, 61] (i.e., they represent a snap-shot of a scene or a bundle of frames).”
Exploring Visual Relationship for Image Captioning (2018, background): “[36] treats visual relationship as the directed edges to connect two object nodes in the scene graph and the relationships are inferred along the processing of constructing scene graph in an iterative way.”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “By observing the potential of exploiting scene graphs in vision tasks, a variety of approaches are proposed to improve the scene graph generation from images [57, 56, 52, 51, 49, 42, 6].”
GQA: a new dataset for compositional question answering over real-world images (2019, background): “[39] 4We expand the original Visual Genome dataset with 5k new scene graphs collected through crowdsourcing.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, methodology): “We train and evaluate our models on Visual Genome, us-ing the publicly released preprocessed data and splits from [42], containing 150 object classes and 50 relation classes, but sample a development set from the training set of 2000 images.”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “To our surprise, we find that IMP, which was one of the earliest scene graph prediction models actually outperforms numerous more recently proposed methods.”
Unbiased Scene Graph Generation From Biased Training (2020, positive): “Predicate Classification Scene Graph Classification Scene Graph Detection Model Fusion Method mR@20 mR@50 mR100 mR@20 mR50 mR100 mR@20 mR50 mR100 IMP+ [62, 6] - 9.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “ven image, and serves us as the machine’s state graph, where objects correspond to states and relations to valid transitions. Multiple models have been proposed for the task of scene graph generation [84, 85, 16, 88]. Here, we largely follow the approaches of Yang et al. [85] and Chen et al. [16] in”
Graph R-CNN for Scene Graph Generation (2018, methodology): “To be consistent with previous work [18,40], we also adopt the two-stage training – we ﬁrst train the object detector and then train the whole model jointly until convergence.”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “In order to generate high-quality scene graphs from images, a series of works explore different directions such as utilizing spatial context [61, 65, 40], graph structure [60, 58, 34], optimization [8], reinforcement learning [36, 51], semi-supervised training [7] or a contrastive loss [66].”
Attend and Interact: Higher-Order Object Interactions for Video Understanding (2017, background): “Visual relationships can also be realized by constructing a scene graph which uses a structured representation for describing object relationships and their attributes [19, 26, 27, 53].”