Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval

By Sounak Dey, Pau Riba, Anjan Dutta, Josep Lladós, Yi-Zhe Song. 2019

212 citations recorded by the source corpus.

In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “For learning the joint embedding space, category-level SBIR typically employs either CNN [11, 12], RNN [65], or Transformer [41] based Siamese networks, accompanied by a triplet-ranking objective [68].”
Sketch-a-Segmenter: Sketch-Based Photo Segmenter Generation (2020, background): “Details conveyed in sketch have been exploited for ﬁne-grained image retrieval [13]–[16] and image synthesis [17]–[19].”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, background): “Research has flourished in recent years, where the main focus has been on addressing the sketch-photo domain gap [33, 16, 44] and data scarcity [5, 3, 10, 35, 12].”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, methodology): “However we find it helpful to project the latent feature through a fully connected layer into 256 dimensional embedding space, and optimise the model through triplet loss [14, 7].”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Following recent stateof-the-art sketch feature extraction pipelines [6, 41], we use soft spatial attention [46] to focus on salient parts of the feature map.”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “In recent years, motivated by the zero-shot setting for supervised photo retrieval [241], zero-shot sketch-based image retrieval (ZSSBIR) has also been studied [65], [216], [242], [243], [244],”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, methodology): “Semantic hashing has been also extensively performed for sketch based image retrieval [180], [167], [209], [146], cross-modal retrieval [192], [205], [183].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Overall, category level SBIR makes use of Siamese networks based on either CNN [11, 12], RNN [53], Transformer [36] or their combinations [11] along with a triplet-ranking objective to learn a joint embedding space.”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, methodology): “The area is split into ﬁne-grained (FG-SBIR) (Yu et al., 2016; Sangkloy et al., 2016; Bhunia et al., 2020) and a zero-shot setting (ZS-SBIR) (Dutta & Akata, 2019; Pandey et al., 2020; Dey et al., 2019).”
Sketchformer: Transformer-Based Representation for Sketched Structure (2020, background): “The generalization of sketch embeddings beyond training classes have also been studied [4, 15], and parameterized for zero-shot learning [9].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, background): “Other related problems such as zero-shot SBIR [9, 10, 24] and sketch-photo hashing [23, 35] are also studied.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, methodology): “Despite starting as a category-level retrieval problem [9, 10, 3, 12], the fine-grained nature of sketches stirred current research focus more towards fine-grained SBIR (FGSBIR) [5, 28] – which aims to retrieve a particular photo based on a query sketch at an intra-category basis.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “sketchbased image retrieval (SBIR) [79,77,36,45,63,58,82,7,30, 13,11] has received a lot of attention.”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “In contrast later works [14, 16] used semantic representation (word2vec) of class labels to learn a joint manifold capable of semantic transfer to unseen categories.”