Graph R-CNN for Scene Graph Generation

By Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, Devi Parikh. 2018

953 citations recorded by the source corpus.

We propose a novel scene graph generation model called Graph R-CNN, that is both effective and efficient at detecting objects and their relations in images. Our model contains a Relation Proposal Network (RePN) that efficiently deals with the quadratic number of potential relations between objects in an image. We also propose an attentional Graph Convolutional Network (aGCN) that effectively captures contextual information between objects and relations. Finally, we introduce a new evaluation metric that is more holistic and realistic than existing metrics. We report state-of-the-art performance on scene graph generation as evaluated using both existing and our proposed metrics.

Subfields: Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

Local Relation Networks for Image Recognition (2019, background): “However, in computer vision, there are few works involving such compositionality in their network architecture [12, 28, 32, 19, 29, 13, 30] .”
Continuous Scene Representations for Embodied AI (2022, methodology): “It is common to build scene graphs for static images [14,32,34,61,62,64].”
A Comprehensive Survey on Graph Neural Networks (2019, background): “Scene graph generation models aim to parse an image into a semantic graph which consists of objects and their semantic relationships [137], [138], [139].”
Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning (2019, background): “It is worth noting that, as Meta R-CNN solely remodels the predictor branches into a meta-learner, it potentially can be extended to a broad range of models [15, 20, 24, 45] in the entire R-CNN family.”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Previous research has proposed plenty of methods for predicting scene graphs on static images [48, 52, 77, 78, 85, 86].”
Unbiased Scene Graph Generation From Biased Training (2020, critical): “Most of the existing methods [62, 60, 7, 26, 68, 53, 64, 10, 43, 59] struggle for better feature extraction networks.”
Learning by Abstraction: The Neural State Machine (2019, background): “Multiple models have been proposed for the task of scene graph generation [84, 85, 16, 88].”
Unified Graph Structured Models for Video Understanding (2021, methodology): “While GNN-based approaches have also been used in scene graph parsing [10, 33, 64, 66] for single images, to our knowledge, they have not been applied to model temporal relations in video.”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, background): “Most of them [17, 23, 24, 36, 42] apply graph reasoning on image regions to learn visual relationships.”
Graph convolutional networks: a comprehensive review (2019, background): “propose an attentional graph convolutional model that can place attention on the reliable edges while dampening the influence of unlikely edges [78].”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “In order to generate high-quality scene graphs from images, a series of works explore different directions such as utilizing spatial context [61, 65, 40], graph structure [60, 58, 34], optimization [8], reinforcement learning [36, 51], semi-supervised training [7] or a contrastive loss [66].”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “[42] reported the results on two more metrics: Predicate Recognition and Phrase Recognition.”
Learning to Generate Scene Graph from Natural Language Supervision (2021, background): “Most previous scene graph methods [52, 28, 62, 27, 53, 4, 46, 45, 65] follow a fully supervised approach, relying on human annotations of object bounding boxes, object categories and their relationships.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “Furthermore, Graph R-CNN [75] attempts to trim the original”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, background): “A variety of papers have considered modeling spatial relations in natural images [7, 22, 37], and scene graph is widely used in the image retrieval tasks [24, 56].”