Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation

By Yikang Li, Wanli Ouyang, Bolei Zhou, Yawen Cui, Jianping Shi, Xiaogang Wang. 2018

284 citations recorded by the source corpus.

Generating scene graph to describe the object interactions inside an image gains increasing interests these years. However, most of the previous methods use complicated structures with slow inference speed or rely on the external data, which limits the usage of the model in real-life scenarios. To improve the efficiency of scene graph generation, we propose a subgraph-based connection graph to concisely represent the scene graph during the inference. A bottom-up clustering method is first used to factorize the entire graph into subgraphs, where each subgraph contains several objects and a subset of their relationships. By replacing the numerous relationship representations of the scene graph with fewer subgraph and object features, the computation in the intermediate stage is significantly reduced. In addition, spatial information is maintained by the subgraph features, which is leveraged by our proposed Spatial-weighted Message Passing (SMP) structure and Spatial-sensitive Relation Inference (SRI) module to facilitate the relationship recognition. On the recent Visual Relationship Detection and Visual Genome datasets, our method outperforms the state-of-the-art method in both accuracy and speed. Code has been made publicly available (https://github.com/yikang-li/FactorizableNet).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Local Relation Networks for Image Recognition (2019, background): “However, in computer vision, there are few works involving such compositionality in their network architecture [12, 28, 32, 19, 29, 13, 30] .”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Given its versatile utility, the task of scene graph prediction has resulted in a series of publications [14, 30, 43, 46, 48, 49, 59, 77, 78, 85] that have explored reinforcement learning [49], structured prediction [16, 40, 70], utilizing object attributes [20, 61], sequential prediction [59], few-”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, methodology): “Like other approaches [26, 25, 37], during training we want the generated scene graph close to the ground-truth scene graph by optimizing the scene graph generation process with object detection loss and relationship classification loss”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, methodology): “In the vision community, we have also seen attempts to construct the so-called scene graph [31, 38] that can describe a visual scene in a similar way to a knowledge graph [44, 45, 65, 67, 69, 77].”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, methodology): “In sub-graph-based scene graph generation, a method of clustering object pairs into graphs by clustering and sharing representations is used [76].”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, methodology): “captioning, while [26, 45] use GCN with attention mechanisms to produce the scene graph from plain images.”
DeepGCNs: Can GCNs Go As Deep As CNNs? (2019, methodology): “This graph is used to detect and segment objects in images, and also to predict semantic relations between object pairs [30, 44, 47, 20].”
PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph (2019, background): “, image retrieval [18], image captioning evaluation [19], sentence-scene graph translation [20], and image-based scene graph prediction [21, 22, 23].”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “It mainly includes chains [68], fully connected graph [21], [23], [27], [31], [81], [82], [83], subgraph [27], and tree [72] structures (see Fig.”
Question-Guided Hybrid Convolution for Visual Question Answering (2018, background): “Convolution Neural Networks (CNN) and Recurrent Neural Networks (RNN) have shown great success in vision and language tasks [12, 39, 46].”
Learning to Generate Scene Graph from Natural Language Supervision (2021, background): “Most previous scene graph methods [52, 28, 62, 27, 53, 4, 46, 45, 65] follow a fully supervised approach, relying on human annotations of object bounding boxes, object categories and their relationships.”
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020, methodology): “Later, Visual Genome [19], a large scale dataset with scene graph annotations on images, gave rise to a line of deep learning based advances on scene graph prediction from images [48, 11, 37, 51, 25, 49, 24, 33].”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “In order to generate high-quality scene graphs from images, a series of works explore different directions such as utilizing spatial context [61, 65, 40], graph structure [60, 58, 34], optimization [8], reinforcement learning [36, 51], semi-supervised training [7] or a contrastive loss [66].”
Learning to Compose Dynamic Tree Structures for Visual Contexts (2018, background): “As shown in Figure 1, two popular structures are chains [57] and fullyconnected graphs [7, 10, 15, 25, 50, 55, 49], where the context is encoded by sequential models such as bidirectional LSTM [19] for chains and CRF-RNN [61] for graphs.”