Knowledge-Embedded Routing Network for Scene Graph Generation

By Tianshui Chen, Weihao Yu, Riquan Chen, Liang Lin. Computer Vision and Pattern Recognition, 2019

425 citations recorded by the source corpus.

To understand a scene in depth not only involves locating/recognizing individual objects, but also requires to infer the relationships and interactions among them. However, since the distribution of real-world relationships is seriously unbalanced, existing methods perform quite poorly for the less frequent relationships. In this work, we find that the statistical correlations between object pairs and their relationships can effectively regularize semantic space and make prediction less ambiguous, and thus well address the unbalanced distribution issue. To achieve this, we incorporate these statistical correlations into deep neural networks to facilitate scene graph generation by developing a Knowledge-Embedded Routing Network. More specifically, we show that the statistical correlations between objects appearing in images and their relationships, can be explicitly represented by a structured knowledge graph, and a routing mechanism is learned to propagate messages through the graph to explore their interactions. Extensive experiments on the large-scale Visual Genome dataset demonstrate the superiority of the proposed method over current state-of-the-art competitors.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/beb335e43da1f518ed7df3a851784925254a6a94

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition (2019, methodology): “Inspired by the current graph propagation works [18, 1, 27, 3], we adopt a gated recurrent update mechanism to propagate message through the graph and learn contextualized node-level features.”
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation (2020, methodology): “It also allows Neural Motifs (NM) to signiﬁcantly outperform KERN [6] on zero-shots (R ZS @100), while being only slightly worse in one of the mR@100 results.”
Learning to Generate Scene Graph from Natural Language Supervision (2021, methodology): “SGDet captures both the localization and classification performance using metrics of Recall@K (R@K) [29, 52] and mean Recall@K (mR@K) [5, 46].”
Learning from the Past: Meta-Continual Learning with Knowledge Embedding for Jointly Sketch, Cartoon, and Caricature Face Recognition (2020, positive): “We introduce the gated mechanism that embeds the knowledge representation to enhance the representation learning, considering suppressing non-informative features and allowing informational features to pass under the guidance of the our knowledge graph, similar to [5], we introduce a gated mechanism”
Spatio-Temporal Dynamic Inference Network for Group Activity Recognition (2021, background): “Many research areas inherently involve the modeling of interactions like trajectory prediction [16, 38, 34], human object interaction [33, 29, 14] and scene graph generation [44, 49, 7].”
Learning by Abstraction: The Neural State Machine (2019, methodology): “Here, we largely follow the approaches of Yang et al. [85] and Chen et al. [16] in conjunction with a variant of the Mask R-CNN object detector [34] proposed by Hu et al. [39].”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “Other works focus on leveraging external information [18] or explicit scene graph [5] to extract features from input images.”
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models (2021, methodology): “Following previous works (Xu et al., 2017; Chen et al., 2019), we use recall@N (R@N) and mean recall@N (mR@N) as the evaluation metrics.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “In contrast, VCTree [72], KERN [20], GPS-Net [185], GB-NET [122], in addition to using visual features, they also use other knowledge (for example, language embedding, statistical information, and counterfactual causality, etc.).”
GPS-Net: Graph Property Sensing Network for Scene Graph Generation (2020, methodology): “Speciﬁcally, GPS-Net outperforms one very recent one-stage model, named KERN [23], by 1.8 % on average at R@50 and R@100 over the three protocols.”
Meta Module Network for Compositional Visual Reasoning (2019, methodology): “The performance gap with NSM [19] is debatable since our model is standalone without relying on well-tuned external scene graph generation model [42, 43, 6].”
Unbiased Scene Graph Generation From Biased Training (2020, positive): “Among all three models and two fusion functions, even the worst TDE performance outperforms previous state-of-theart methods [53, 6] by a large margin on RR mR@K.”
Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “[2] employed a less explicit way to incorporate the frequency prior within the process of entity and predicate classification.”