Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval

By Ayan Kumar Bhunia, Yongxin Yang, Timothy M. Hospedales, Tao A. Xiang, Yi-Zhe Song. 2020

128 citations recorded by the source corpus.

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least number of strokes possible. We further propose an on-the-fly design that starts retrieving as soon as the user starts drawing. To accomplish this, we devise a reinforcement learning based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode. Additionally, we introduce a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval. We achieve superior early-retrieval efficiency over state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “The advocate of sketches is largely motivated by the line of work examining human-centric characteristics of sketches in many parallel applications – notably image re-a trieval [12], where the ﬁne-grained nature of sketches is used to successfully conduct instance-level retrieval [2, 6, 46, 14, 11].”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, background): “[179] developed an on-the-fly retrieval setup via reinforcement learning that begins retrieving photos”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, background): “The area is split into fine-grained (FG-SBIR) (Bhunia et al., 2020; Sangkloy et al., 2016; Yu et al., 2016) and a zero-shot setting (ZS-SBIR) (Dey et al.”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, methodology): “[5] employed reinforcement learning in an early retrieval scenario.”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, background): “However we find it helpful to project the latent feature through a fully connected layer into 256 dimensional embedding space, and optimise the model through triplet loss [14, 7].”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, methodology): “Following [9] we use 6, 051 and 1, 800 respectively for training and the rest for testing.”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Preliminary analysis: The performance barrier due to irrelevant strokes gets noticed under on-the-fly FG-SBIR [7] setup.”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “[8] employed reinforcement learning in an early retrieval scenario.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, methodology): “Cross-category generalisation [26] and on-the-fly retrieval setup [5] are more recent additions to existing FG-SBIR literature.”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “the sequential [13], abstract [26], and stroke-wise [1] characteristics of sketches, either separately or in combination [33, 48].”
Deep Learning for Free-Hand Sketch: A Survey (2020, methodology): “Similarly, the challenges of these tasks are also mainly due to the large domain gap between free-hand sketches and other modalities.”