Visual Relationship Detection with Language Priors

By Cewu Lu, Ranjay Krishna, Michael S. Bernstein, Li Fei-Fei. 2016

1,248 citations recorded by the source corpus.

Visual relationships capture a wide variety of interactions between pairs of objects in images (e.g. “man riding bicycle” and “man pushing bicycle”). Consequently, the set of possible relationships is extremely large and it is difficult to obtain sufficient training examples for all possible relationships. Because of this limitation, previous work on visual relationship detection has concentrated on predicting only a handful of relationships. Though most relationships are infrequent, their objects (e.g. “man” and “bicycle”) and predicates (e.g. “riding” and “pushing”) independently occur more frequently. We propose a model that uses this insight to train visual models for objects and predicates individually and later combines them together to predict multiple relationships per image. We improve on prior work by leveraging language priors from semantic word embeddings to finetune the likelihood of a predicted relationship. Our model can scale to predict thousands of types of relationships from a few examples. Additionally, we localize the objects in the predicted relationships as bounding boxes in the image. We further demonstrate that understanding relationships can improve content based image retrieval.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

MERLOT: Multimodal Neural Script Knowledge Models (2021, neutral): “To probe our model’s ability to do out-of-the-box commonsense reasoning over events in images, we next consider the Visual Storytelling dataset [50, 74].”
Logic Tensor Networks for Semantic Image Interpretation (2017, background): “In [Lu et al., 2016] a similar neural network is proposed for the very same task but with a more sophisticated language model.”
Scene Graph Generation by Iterative Message Passing (2017, methodology): “Most methods have focused on making local predictions of object relationships [26, 34], which essentially simplify the scene graph generation problem into independently predicting relationships between pairs of objects.”
Modeling Relationships in Referential Expressions with Compositional Modular Networks (2016, methodology): “Recently work by [17] trains detectors based on RCNN [8] and uses a linguistic prior to detect visual relationships.”
Exploring Visual Relationship for Image Captioning (2018, background): “In the literature, it is well recognized that reasoning such visual relationships is crucial to a richer semantic understanding [19,23] of the visual world.”
Image Generation from Scene Graphs (2018, methodology): “Scene graphs have been used for image retrieval [22] and to evaluate image captioning [1]; some work converts sentences to scene graphs [47] or predicts grounded scene graphs for images [32, 36, 57, 58].”
Weakly-Supervised Learning of Visual Relations (2017, methodology): “We evaluate our method on the Visual Relation-ship Detection dataset [31] following the original experimental setup.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, methodology): “Scene Graph Methods Several works have explored the role of priors by incorporating background language statistics [26] or by attempting to preprocess scene graphs [49].”
Zero-Shot Object Detection (2018, methodology): “Recall has also been used in prior work on detecting visual relationships [33] where it is infeasible to annotate all possible instances.”
Detecting and Recognizing Human-Object Interactions (2017, background): “In contrast, in an open-world recognition setting, evaluating precision is not feasible, resulting in recall-based evaluation, as in [23].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, methodology): “We refer the reader to the paper that introduced these tasks for more details [52].”
A Domain Based Approach to Social Relation Recognition (2017, background): “Social relation is different with the visual relation of “subject-predicate-object” [33, 34], but it is derived from human social behaviors.”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “[32] as Zero-Shot Recall@K and was firstly evaluated on VG dataset in this paper, which only reports the R@K of those subject-predicate-object triplets that have never been observed in the training set.”
Graph R-CNN for Scene Graph Generation (2018, methodology): “A number of approaches have been proposed for the detection of both objects and their relationships [2,17–19, 23, 26 , Most scene graph generation methods also include some mechanisms for context propagation and reasoning over a candidate scene graph in order to reﬁne the ﬁnal labeling.”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…of vision tasks like image captioning (Hu et al., 2016; Rohrbach et al., 2016; Lu et al., 2018), visual question answering (Lu et al., 2016b; Agrawal et al., 2018), relationship detection (Lu et al., 2016a; Ma et al., 2018) and referral expression (Nagaraja et al., 2016; Gavrilyuk et al., 2018).”
Large-Scale Visual Relationship Understanding (2018, methodology): “The only consistent known setting is when approaches use the publicly available proposals provided by [21], like [21, 27, 41].”