A Zero-Shot Framework for Sketch-based Image Retrieval

By Sasi Kiran Yelamarthi, Shiva Krishna Reddy Malay, Ashish Mishra, Anurag Mittal. 2018

217 citations recorded by the source corpus.

Sketch-based image retrieval (SBIR) is the task of retrieving images from a natural image database that correspond to a given hand-drawn sketch. Ideally, an SBIR model should learn to associate components in the sketch (say, feet, tail, etc.) with the corresponding components in the image having similar shape characteristics. However, current evaluation methods simply focus only on coarse-grained evaluation where the focus is on retrieving images which belong to the same class as the sketch but not necessarily having the same shape characteristics as in the sketch. As a result, existing methods simply learn to associate sketches with classes seen during training and hence fail to generalize to unseen classes. In this paper, we propose a new benchmark for zero-shot SBIR where the model is evaluated on novel classes that are not seen during training. We show through extensive experiments that existing models for SBIR that are trained in a discriminative setting learn only class specific mappings and fail to generalize to the proposed zero-shot setting. To circumvent this, we propose a generative approach for the SBIR task by proposing deep conditional generative models that take the sketch as an input and fill the missing information stochastically. Experiments on this new benchmark created from the “Sketchy” dataset, which is a large-scale database of sketch-photo pairs demonstrate that the performance of these generative models is significantly better than several state-of-the-art approaches in the proposed zero-shot framework of the coarse-grained SBIR task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “Contemporary research on this category is also directed towards zero-shot SBIR [12, 67, 44] and binary hash-code embedding [30, 49].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “[65], with an aim to minimise sketch-photo domain gap by approximating photo features from given sketches via image-to-image translation, thus aligning sketch-photo features jointly to generalise onto unseen classes.”
Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning (2019, background): “[36], zero-shot learning [42, 1, 15], visual tracking [19, 31] and person re-identiﬁcation [41, 9].”
TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, background): “Transfer learning [29] has enjoyed the prosperity recently in a number of areas [1, 6, 7, 18].”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “One of the major shortcomings of the prior work on ZS-SBIR is that sketch-image is retrieved after learning a mapping from an input sketch to an output image using a training set of labelled aligned pairs [23].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Further efforts have been made through zero-shot SBIR [13, 56] for cross-category generalisation, and employing binary hash-code embedding [29, 43] to reduce the computational complexity.”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, methodology): “In the computer vision (CV) literature, deep metric learning (DML) has been used to great effect for learning visual representations [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34].”
Generalising Fine-Grained Sketch-Based Image Retrieval (2019, methodology): “We follow the same dataset split as [35] and partition Sketchy into 104 train and 21 test categories to ensure the test ones are not present in 1000 ImageNet Challenge classes [23].”
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “[36] resort to a deep conditional generative model, where a sketch is taken as input and learned to generate its photo features by stochastically filling the missing information.”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “Thus, common approaches are to train ranking [65], [251] or generative [245], [246] models for retrieval, which are enhanced and”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, background): “Contemporary research directions also include zero-shot SBIR [59, 10] where a model aims to generalise across disjoint training and testing classes [12], alleviating annotation costs.”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Whereas zero-shot SBIR [47, 6, 9, 24] asks models to generalise across disjoint training and testing classes in order to alleviate annotation costs.”
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “Thus, in Table 1, we reported our model’s performance on this careful split proposed by [16], which only use classes that are not present in the 1000 classes of ImageNet as testing.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “With the ever rising popularity of touch screen devices, sketch-based image retrieval (SBIR) has witnessed significant interest within the vision community [5, 34, 35, 50, 11, 45].”