Semantic Image Retrieval via Active Grounding of Visual Situations
By Max H. Quinn, Erik Conser, Jordan M. Witte, Melanie Mitchell. 2017
10 citations recorded by the source corpus.
We describe a novel architecture for semantic image retrieval—in particular, retrieval of instances of visual situations. Visual situations are concepts such as “a boxing match,” “walking the dog,” “a crowd waiting for a bus,” or “a game of pingpong,” whose instantiations in images are linked more by their common spatial and semantic structure than by low-level visual similarity. Given a query situation description, our architecture—called Situate—learns models capturing the visual features of expected objects as well the expected spatial configuration of relationships among objects. Given a new image, Situate uses these models in an attempt to ground (i.e., to create a bounding box locating) each expected component of the situation in the image via an active search procedure. Situate uses the resulting grounding to compute a score indicating the degree to which the new image is judged to contain an instance of the situation. Such scores can be used to rank images in a collection as part of a retrieval system. In the preliminary study described here, we demonstrate the promise of this system by comparing Situate’s performance with that of two baseline methods, as well as with a related semantic image-retrieval system based on “scene graphs.”
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Oron Ashual
- Lior Wolf
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Jitendra Malik
Citations in the corpus
- Object-Centric Image Generation from Layouts (2020, neutral): “Despite similar interest in scene graph to image retrieval (Johnson et al. 2015; Quinn et al. 2018), and the large improvements in text-to-image synthesis resulting from the DAMSM (Xu et al.”
- Specifying Object Attributes and Relations in Interactive Scene Generation (2019, methodology): “A scene graph representation is often used for retrieval based on text [10, 17], and a few datasets include this information, e.g., COCO-stuff [2] and the visual Figure 2.”