Neural Motifs: Scene Graph Parsing with Global Context

By Rowan Zellers, Mark Yatskar, Sam Thomson, Yejin Choi. 2017

1,179 citations recorded by the source corpus.

We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find that there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at least two relations. Our analysis motivates a new baseline: given object detections, predict the most frequent relation between object pairs with the given labels, as seen in the training set. This baseline improves on the previous state-of-the-art by an average of 3.6% relative improvement across evaluation settings. We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7.1% relative gain. Our code is available at github.com/rowanz/neural-motifs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “Last, annotation artifacts are also a re-curring problem for vision datasets such as Visual Genome (Zellers et al., 2018) and Visual QA (Jabri et al., 2016); recent work was done to create a more challenging VQA dataset by annotating complementary image pairs (Goyal et al., 2016).”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Various methods have been proposed to parse scene graphs from images [26] [27] and texts [28] [29] [30].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “It is worse on getting both right (‘action’), we suspect that this might be due to priors (motifs) between noun and verb [127].”
The Open Images Dataset V4 (2018, methodology): “We compute two frequency baselines inspired by (Zellers et al., 2018).”
Continuous Scene Representations for Embodied AI (2022, background): “It is common to build scene graphs for static images [14,32,34,61,62,64].”
Troubling Trends in Machine Learning Scholarship (2018, background): “ined coreference errors). In addition, simpler neural networks and linear classiﬁers outperformed complicated neural architectures that had previously been evaluated on this task. In the same spirit, [80] analyzes and constructs a strong baseline for the Visual Genome Scene Graphs dataset. 3.3 Mathi”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “By observing the potential of exploiting scene graphs in vision tasks, a variety of approaches are proposed to improve the scene graph generation from images [57, 56, 52, 51, 49, 42, 6].”
What Does BERT with Vision Look At? (2020, background): “Extensive experiments on four vision-and-language datasets (Goyal et al., 2017; Zellers et al., 2018; Suhr et al., 2019; Plummer et al., 2015) verify that pre-trained VisualBERT exceeds all comparable baselines signiﬁcantly.”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Previous research has proposed plenty of methods for predicting scene graphs on static images [48, 52, 77, 78, 85, 86].”
Unbiased Scene Graph Generation From Biased Training (2020, positive): “SGG [62, 69] has received increasing attention in computer vision community, due to the potential revolution that would be brought to down-stream visual reasoning tasks [49, 65, 22, 17].”
Learning by Abstraction: The Neural State Machine (2019, background): “Multiple models have been proposed for the task of scene graph generation [84, 85, 16, 88].”
Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, background): “The evidence lies in the fact that visual contexts will emerge in higher-level layers of CNN when training image classifiers [66, 68], which essentially serve as a feature map backbone for modern visual detection that highly relies on contexts, such as Fast R-CNN [14] and SSD [31].”
Learning to Collocate Neural Modules for Image Captioning (2019, background): “(1) as the input because these kinds of features are widely applied as the input for successful relationship detection [61, 59].”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, neutral): “Most recent approaches achieve this goal by learning classiﬁers that predict relations based on different types of features of the object pairs (Xu et al. 2017; Yu et al. 2017; Zellers et al. 2018; Zhang et al. 2019a; 2019b).”
Knowledge-Embedded Routing Network for Scene Graph Generation (2019, methodology): “Although previous studies [6, 33] have also taken notice of the statistical knowledge, they merely implicitly mine this information by iterative message propagation between relationships and objects [30] or by encoding the global context of objects and relationships [33].”
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020, background): “Later, Visual Genome [19], a large scale dataset with scene graph annotations on images, gave rise to a line of deep learning based advances on scene graph prediction from images [48, 11, 37, 51, 25, 49, 24, 33].”
Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “In Scene Graph Generation (SGG), for instance, external knowledge bases [7] and dataset statistics [2,34] have been utilized to improve the accuracy of entity (object) and predicate (relation) recognition.”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, methodology): “For Motif Freq [65], PredCLS-R @20 increases from 65.1% to 65.2% slightly probably due to the relatively simple feature representation.”