Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

By Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao A. Xiang, Yi-Zhe Song. 2020

61 citations recorded by the source corpus.

Sketch as an image search query is an ideal alternative to text in capturing the fine-grained visual details. Prior successes on fine-grained sketch-based image retrieval (FG-SBIR) have demonstrated the importance of tackling the unique traits of sketches as opposed to photos, e.g., temporal vs. static, strokes vs. pixels, and abstract vs. pixel-perfect. In this paper, we study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail -- a person typically sketches up to various extents of detail to depict an object. This hierarchical structure is often visually distinct. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels. In particular, features from a sketch and a photo are enriched using cross-modal co-attention, coupled with hierarchical node fusion at every level to form a better embedding space to conduct retrieval. Experiments on common benchmarks show our method to outperform state-of-the-arts by a significant margin.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “Moreover, due to its highly abstract nature, the same object may be sketched in various ways under unique user-styles [51, 46], and with varied levels of detail [45].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “[46] discovered cross-modal hierarchy in sketches, Bhunia et al.”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, background): “2) [52, 54-55, 175-180] Fine-Grained Image Retrieval Future Directions of FGIA”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, methodology): “It adapts using inner loop updates across retrieval tasks over categories in SBIR and over instances in FG-SBIR frameworks”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “31% without any complicated joint-training of Semi-Sup [4], costly hierarchical paired embedding of Cross-Hier [40], or meta-learning cumbersome feature transformation layer of StyleMeUp [41].”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, methodology): “In particular, the FGSBIR model requires rasterized sketch-images to obtain the sketch embedding, as performance can collapse on using sketch-coordinate instead [5, 33].”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, background): “Sketch Representation Learning: Learning good sketch representations benefits a variety of sketch-specific problems like classification [64], retrieval [62], scene understanding [38], sketch based image retrieval [7, 45, 46, 14, 16, 4, 53, 52], generative sketch modelling [57, 22, 6] etc.”