Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval

By Sebastian Schuster, Ranjay Krishna, Angel X. Chang, Li Fei-Fei, Christopher D. Manning. 2015

372 citations recorded by the source corpus.

Semantically complex queries which include attributes of objects and relations between objects still pose a major challenge to image retrieval systems. Recent work in computer vision has shown that a graph-based semantic representation called a scene graph is an effective representation for very detailed image descriptions and for complex queries for retrieval. In this paper, we show that scene graphs can be effectively created automatically from a natural language scene description. We present a rule-based and a classifierbased scene graph parser whose output can be used for image retrieval. We show that including relations and attributes in the query graph outperforms a model that only considers objects and that using the output of our parsers is almost as effective as using human-constructed scene graphs (Recall@10 of 27.1% vs. 33.4%). Additionally, we demonstrate the general usefulness of parsing to scene graphs by showing that the output can also be used to generate 3D scenes.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Various methods have been proposed to parse scene graphs from images [26] [27] and texts [28] [29] [30].”
Object-Centric Image Generation from Layouts (2020, background): “2015), as well as in sentence to scene graph (Schuster et al. 2015) and image to scene graph prediction (Lu et al.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “We use a syntactic dependency parser to extract programs and concepts from language (Andreas et al., 2016; Schuster et al., 2015).”
Image Generation from Scene Graphs (2018, methodology): “Scene graphs have been used for image retrieval [22] and to evaluate image captioning [1]; some work converts sentences to scene graphs [47] or predicts grounded scene graphs for images [32, 36, 57, 58].”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “We use the scene graph parser provided by [1] for scene graphs G from sentences, where a syntactic dependency tree is built by [18] and then a rule-based method [39] is applied for transforming the tree to a scene graph.”
Visual Relationship Detection with Language Priors (2016, methodology): “Finally, they have been used to generate indoor images from sentences [39] and to improve image search [8,40].”
Evaluation of Text Generation: A Survey (2020, methodology): “A scene graph (Schuster et al., 2015) encodes objects, attributes, and relationships detected in image captions, representing an image in a skeleton form, as shown in Figure 3.2.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “PICE. SPICE (Semantic Propositional Image Caption Evaluation) [3] is a new caption evaluation metric based on semantic concept. It is based on a graph-based semantic representation called scene-graph [63, 123]. This graph can extract the information of different objects, attributes and their relatio”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Progress in image-based scene graph prediction has cascaded to improvements across multiple Computer Vision tasks, including image captioning [2], image retrieval [36, 64], visual question answering [35], relationship modeling [41] and image generation [34].”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “from the previous image retrieval with scene graph [18, 48], because the latter still consider the images as visual features but not SGs.”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “The applications include image retrieval [46], image captioning [1, 45], VQA [51, 25] and image generation [24, 19].”
Re-evaluating Automatic Metrics for Image Captioning (2016, background): “It is based on the agreement of the scenegraph tuples (Johnson et al., 2015; Schuster et al., 2015) of the candidate sentence and all reference sentences.”
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos (2017, background): “This includes recent progress in scene graphs [13, 20, 51, 64], storylines [1, 14, 17, 31, 53], and action understanding [10, 44, 55].”
The Promise of Premise: Harnessing Question Premises in Visual Question Answering (2017, methodology): “Originally defined as a metric for image captioning, SPICE transforms a sentence into a scene graph using the Stanford Scene Graph Parser (Schuster et al., 2015) and then extracts”
Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation (2017, methodology): “We utilize the scene graph parser proposed in [28] to parse sentences into sets of 〈subj, pred, obj〉 triplets, and we compute the conditional probabilities of predicates based on these triplets.”