Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval

By Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, Ling Shao. 2017

298 citations recorded by the source corpus.

Free-hand sketch-based image retrieval (SBIR) is a specific cross-view retrieval task, in which queries are abstract and ambiguous sketches while the retrieval database is formed with natural images. Work in this area mainly focuses on extracting representative and shared features for sketches and natural images. However, these can neither cope well with the geometric distortion between sketches and images nor be feasible for large-scale SBIR due to the heavy continuous-valued distance computation. In this paper, we speed up SBIR by introducing a novel binary coding method, named Deep Sketch Hashing (DSH), where a semi-heterogeneous deep architecture is proposed and incorporated into an end-to-end binary coding framework. Specifically, three convolutional neural networks are utilized to encode free-hand sketches, natural images and, especially, the auxiliary sketch-tokens which are adopted as bridges to mitigate the sketch-image geometric distortion. The learned DSH codes can effectively capture the cross-view similarities as well as the intrinsic semantic correlations between different categories. To the best of our knowledge, DSH is the first hashing work specifically designed for category-level SBIR with an end-to-end deep architecture. The proposed DSH is comprehensively evaluated on two large-scale datasets of TU-Berlin Extension and Sketchy, and the experiments consistently show DSHs superior SBIR accuracies over several state-of-the-art methods, while achieving significantly reduced retrieval time and memory footprint.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “Contemporary research on this category is also directed towards zero-shot SBIR [12, 67, 44] and binary hash-code embedding [30, 49].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Further efforts have been made through zero-shot SBIR [13, 56] for cross-category generalisation, and employing binary hash-code embedding [29, 43] to reduce the computational complexity.”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “Category-level SBIR is evaluated similar to [28] using mean average precision (mAP@all) and precision considering top 200 (P@200) retrievals.”
Sketch-Guided Object Localization in Natural Images (2020, background): “Despite the problem of sketch-guided object localization has a relation to the well-studied sketch-based image retrieval (SBIR) problem [5,14,20,25,28,30,38,40], these are distinctively different in terms of objectives.”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, methodology): “(2017) [207] have introduced a semi-heterogeneous deep”
Fine-Grained Image Retrieval: the Text/Sketch Input Dilemma (2017, background): “Fine-grained Sketch-based Image Retrieval The traditional category-level SBIR problem has been well studied [2, 9, 15, 24, 27, 35].”
A Zero-Shot Framework for Sketch-based Image Retrieval (2018, methodology): “We follow the training methodology described in [7, 26, 40] closely.”
Zero-Shot Sketch-Image Hashing (2018, methodology): “, Sketchy [52] and TU-Berlin [15], with extended images obtained from [38].”
DeepStore: In-Storage Acceleration for Intelligent Queries (2019, background): “, the keys in transactional database and key-value store), and can comprise of images/image-patches [92, 96, 106], sketch [70], color map [91], context map [98, 99], music [72, 97], and many more, intelligent query systems also leverage DNNs to automatically extract the features from query input to ”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, methodology): “Existing SBIR models [30, 8, 47, 35, 44] focus primarily on bridging the gap between the photo and sketch modalities.”
Unsupervised Deep Hashing with Similarity-Adaptive and Discrete Optimization (2018, background): “[28] design a novel three-network framework for the sketchbased image retrieval (SBIR) task.”
Generalising Fine-Grained Sketch-Based Image Retrieval (2019, background): “This has been actively studied in recent years due to its challenge as a vision problem, and commercial relevance [19, 36, 24, 20, 41].”
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “Realising the need of large-scale SBIR, some researchers have proposed a variant of cross-modal hashing framework for the same [16, 36], which also showed promising results in SBIR scenario.”
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval (2018, methodology): “DSH-Sketch [16] is proposed to specifically target on modeling the sketch-photo cross-domain relations with a semi-heterogeneous network.”
Enhancing Sketch-Based Image Retrieval by CNN Semantic Re-ranking (2020, methodology): “For DSH [40], the public caffemodel for Sketchy dataset is used to get the 128-bit hash codes for Sketchy, and the public caffemodel for TU-Berlin Extension dataset is used to get the 128-bit hash codes for TU-Berlin Extension.”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Category-level SBIR: Category-level sketch-photo retrieval is now well studied [3, 43, 2, 6, 5, 47, 39, 9, 24, 4, 23].”
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “We evaluated SAKE on two large-scale sketchphoto datasets: TU-Berlin [5] and Sketchy [33] with extended images obtained from [44, 21].”
Cross-domain Generative Learning for Fine-Grained Sketch-Based Image Retrieval (2017, methodology): “It has received increasing research attention recently [12, 14, 19, 22, 23, 27], due to its potential in commercial applications such as searching online product catalogues for shoes, furniture, and handbags by ﬁnger-sketching on a touchscreen device.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “[27] extended the dataset by adding 60, 502 photos yielding in total 73, 002 images.”
Creative Sketch Generation (2020, background): “, 2018), sketch-based image retrieval (Yu et al., 2016; Liu et al., 2017; Ribeiro et al., 2020) and generation (Gao et al.”