Detecting Visual Relationships with Deep Relational Networks

By Bo Dai, Yuqi Zhang, Dahua Lin. 2017

527 citations recorded by the source corpus.

Relationships among objects play a crucial role in image understanding. Despite the great success of deep learning techniques in recognizing individual objects, reasoning about the relationships among objects remains a challenging task. Previous methods often treat this as a classification problem, considering each type of relationship (e.g. ride) or each distinct visual phrase (e.g. person-ride-horse) as a category. Such approaches are faced with significant difficulties caused by the high diversity of visual appearance for each kind of relationships or the large number of distinct visual phrases. We propose an integrated framework to tackle this problem. At the heart of this framework is the Deep Relational Network, a novel formulation designed specifically for exploiting the statistical dependencies between objects and their relationships. On two large data sets, the proposed method achieves substantial improvement over state-of-the-art.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “It is common to build scene graphs for static images [14,32,34,61,62,64].”
Exploring Visual Relationship for Image Captioning (2018, methodology): “[5,19] directly learn the visual features for relationship prediction based on additional union bounding boxes which cover object and subject together.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, methodology): “Furthermore, recent graph-propagation methods were applied but converge quickly and bottle neck through edges, significantly limiting information exchange [47, 25, 6, 23].”
Towards Diverse and Natural Image Descriptions via a Conditional GAN (2017, methodology): “Such methods ﬁrst detect visual concepts ( e.g . object categories, relationships, and attributes) using CRFs [2, 6, 12], SVMs [16], or CNNs [5,17], then generate descriptions thereon using simple methods, such as sentence templates [12, 16], or by retrieving relevant sentences from existing data [5”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Given its versatile utility, the task of scene graph prediction has resulted in a series of publications [14, 30, 43, 46, 48, 49, 59, 77, 78, 85] that have explored reinforcement learning [49], structured prediction [16, 40, 70], utilizing object attributes [20, 61], sequential prediction [59], few-”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “Most of the existing methods [62, 60, 7, 26, 68, 53, 64, 10, 43, 59] struggle for better feature extraction networks.”
Graph R-CNN for Scene Graph Generation (2018, methodology): “A number of approaches have been proposed for the detection of both objects and their relationships [2,17–19, 23, 26 , Most scene graph generation methods also include some mechanisms for context propagation and reasoning over a candidate scene graph in order to reﬁne the ﬁnal labeling.”
Attend and Interact: Higher-Order Object Interactions for Video Understanding (2017, background): “Another approach which has been widely used with images is by pairing all possible object candidates (or subject-object pairs) [7, 8, 17, 39, 58].”
Localizing Moments in Video with Natural Language (2017, background): “of work as we speciﬁcally study temporal prepositions and how they refer to video. Modeling Visual Relationships. A variety of papers have considered modeling spatial relationships in natural images (Dai et al., 2017; Hu et al., 2017; Peyre et al., 2017; Plummer et al., 2017). Our approach is analog”
Scene Graph Generation from Objects, Phrases and Region Captions (2017, background): “Recently, researchers pay more attention to general visual relationship detection [28, 40, 32, 44, 8, 50, 51] .”
Video Visual Relation Detection (2017, methodology): “[5, 42] particularly used coordinate or binary mask based features to enhance the performance of detecting spatial relation.”
Pixels to Graphs by Associative Embedding (2017, methodology): “For example: incorporating vision and language when reasoning over a pair of objects [18]; using message-passing RNNs to process a set of proposed object boxes [26]; predicting over triplets of bounding boxes that corresponding to proposals for a subject, phrase, and object [15]; using reinforcement”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, background): “A variety of papers have considered modeling spatial relations in natural images [7, 22, 37], and scene graph is widely used in the image retrieval tasks [24, 56].”
LinkNet: Relational Embedding for Scene Graph (2018, unknown): “This includes detection of human-object interactions [7, 3], localization of proposals from natural language expressions [12], or the more general tasks of visual relationship detection [17, 25, 38, 5, 19, 37, 34, 41] and scene graph generation [33, 18, 35, 22].”
Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation (2017, methodology): “Unlike [17, 18, 2], which do not use linguistic knowledge explicitly, we focus on predicting predicates using the linguistic knowledge that models correlations between predicates and 〈subj, obj〉 pairs, especially for the long-tail relationships.”
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection (2018, background): “A number of recent work addresses the problem of detecting visual relationship [1, 7, 19, 22, 23, 32, 43, 44] and generating scene graph [25, 39, 42].”
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships (2020, methodology): “Inspired by image captioning methods which seek to use scene graph and object relationships [10, 22, 42] , we propose to fully leverage the spatial relationship between close-by objects instead of individually focusing on information-constrained occluded objects.”
Unified Graph Structured Models for Video Understanding (2021, methodology): “While GNN-based approaches have also been used in scene graph parsing [10, 33, 64, 66] for single images, to our knowledge, they have not been applied to model temporal relations in video.”
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation (2018, methodology): “All the experiments are performed on VG-MSDN [4] as it is larger than VRD [1] to eliminate overfitting and contains more predicate categories than VG-DR-Net [6].”
Learning to Detect Human-Object Interactions (2017, background): “As a result, follow-up works [2, 21, 20, 37, 38] Figure 2: Generating human-object proposals from human and object detections.”