SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval

By Peng Xu, Yongye Huang, Tongtong Yuan, Kaiyue Pang, Yi-Zhe Song, Tao A. Xiang, Timothy M. Hospedales, Zhanyu Ma, Jun Guo. 2018

125 citations recorded by the source corpus.

We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and Abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and Abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “For learning the joint embedding space, category-level SBIR typically employs either CNN [11, 12], RNN [65], or Transformer [41] based Siamese networks, accompanied by a triplet-ranking objective [68].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Overall, category level SBIR makes use of Siamese networks based on either CNN [12, 13], RNN [54], Transformer [37] or their combinations [12] along with a triplet-ranking objective to learn a joint embedding space.”
Sketch-Guided Object Localization in Natural Images (2020, background): “Since the time complexity of such methods for large-scale SBIR problem is significantly high, researchers have proposed hashing models, such as [20,28,35,40], which significantly reduces the retrieval time.”
Sketchformer: Transformer-Based Representation for Sketched Structure (2020, methodology): “In particular, long-short term memory (LSTM) networks have shown signiﬁcant promise in learning search embeddings [32, 5] due to their ability to model higher-level structure and temporal order versus convolutional neural networks (CNNs) on rasterized sketches [3, 18, 6, 22].”
Deep Learning for Free-Hand Sketch: A Survey (2020, methodology): “proposed the sketch hashing network SketchMate [19], where the backbone is a CNN-RNN dual branch network, using CNN to extract abstract visual concepts and RNN to model human temporal stroke order.”
LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search (2019, methodology): “In our approach we adopt the a vector representation for sketches, building upon the SketchRNN variational auto-encoder of Eck et al . previously applied to blend [14] and match [37] sketches with sketches.”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “the sequential [13], abstract [26], and stroke-wise [1] characteristics of sketches, either separately or in combination [33, 48].”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, methodology): “Following [57], we use 10k sketches for each category with 8:1:1 split for meta-training, meta-test and evaluation respectively.”
Multimodal Learning With Transformers: A Survey (2022, background): “language [27], sign language [28], [29], [30], point cloud [31], symbolic knowledge (graph) [32], [33], multimodal knowledge graph [34], sketch drawing [35], [36], [37], [38], 3D object/scene [39], [40], [41], document [42], [43], programming code [44] and Abstract Syntax Tree (AST) – a kind of grap”