Image retrieval using scene graphs

By Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, Li Fei-Fei. 2015

1,234 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “Understanding detailed semantics depicted in an image is critical for visual understanding (Johnson et al., 2015) and prior studies show that modeling such semantics can beneﬁt visual-an-language models.”
A Multimodal Framework for the Detection of Hateful Memes (2020, background): “Recent developments in the field, such as ERNIE-ViL [37], a multimodal model that incorporates structured knowledge from scene graphs [13] in its pretraining tasks, is another interesting avenue for future work.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, background): “However, based on randomly-masking and predicting the sub-words, current models do not distinguish common words and words describing the detailed semantics [13], e.”
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, background): “A scene can also be represented by a scene graph [17, 21], where nodes are objects annotated with attributes and edges connect spatially related objects.”
Object-Centric Image Generation from Layouts (2020, background): “2019) and retrieval (Johnson et al. 2015), as well as in sentence to scene graph (Schuster et al.”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “th several types of focused ﬁll-in-the-blank descriptions (360k in total), some referring to attributes and actions of speciﬁc people and object instances, and some referring to the image as a whole. Johnson et al. (2015b) is another notable work concerned with grounding of semantic scene descriptio”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “We have developed and care-fully reﬁned a robust question engine , leveraging content : information about objects, attributes and relations provided through Visual Genome Scene Graphs [17], along with structure : a newly-created extensive linguistic grammar which couples hundreds of structural patte”
SPICE: Semantic Propositional Image Caption Evaluation (2016, methodology): “Recent work has demonstrated scene graphs to be a highly eﬀective representation for performing complex image retrieval queries [15,16], and we demonstrate similar advantages when using this representation for caption evaluation.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “Scene graph based relationship modeling, using a structured representation for describing object relationships and their attributes [35, 43, 44, 74] has been extensively studied.”
Grounding of Textual Phrases in Images by Reconstruction (2015, methodology): “[14] uses Conditional Random Field to ground the specifically designed ”scene graph” query in the image.”
The More You Know: Using Knowledge Graphs for Image Classification (2016, background): “There has also been some work using a knowledge base for image retrieval [12] or answering visual queries [39], but these works are focused on building and then querying knowledge bases rather than using existing knowledge bases as side information for some vision task.”
Continuous Scene Representations for Embodied AI (2022, background): “Towards building such representations, scene graphs [2, 29, 65] are a candidate as they provide a compact and explicit description of a scene.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “We build a synthetic dataset for image retrieval and adopt a DSL from scene graph–based image retrieval (Johnson et al., 2015).”
Scene Graph Generation by Iterative Message Passing (2017, methodology): “In comparison, scene graphs [18] offer explicit grounding of visual concepts, avoiding referential uncertainty in text-based representation.”
Image Generation from Scene Graphs (2018, methodology): “Scene graphs have been used for image retrieval [22] and to evaluate image captioning [1]; some work converts sentences to scene graphs [47] or predicts grounded scene graphs for images [32, 36, 57, 58].”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “image, which includes the knowledge of present objects, their attributes, and pairwise relationships. Thus, the scene graph can provide a beneﬁ- cial prior for other vision tasks like image retrieval [13], VQA [39], and image generation [11]. By observing the potential of exploiting scene graphs in ”
Visual Relationship Detection with Language Priors (2016, methodology): “.15 7.70 11.5 20 Visual Phrases [6] 8.72 18.12 28.04 12 Our Model 10.82 30.02 47.00 4 truth only if it was selected by both annotators. We evaluate performance using R@1, R@5 and R@10 and median rank [8]. For comparison, we use three image descriptors that are commonly used in image retrieval: CNN [”
Weakly-Supervised Learning of Visual Relations (2017, background): “Triplet learning has been addressed in various tasks such as mining typical relations (knowledge extraction) [6, 42, 51, 53], reasoning [20, 34, 44], object detection [16, 43], image retrieval [22] or fact retrieval [10].”
GQA: a new dataset for compositional question answering over real-world images (2019, background): “Our starting point in creating the GQA dataset is the Visual Genome Scene Graph annotations3 [20] that cover 113k images from COCO [23] and Flickr [37].4 The scene graph serves as a formalized representation of the image: each node denotes an object, a visual entity within the image, like a person, ”