Zero-Shot Sketch-Image Hashing

By Yuming Shen, Li Liu, Fumin Shen, Ling Shao. 2018

172 citations recorded by the source corpus.

Recent studies show that large-scale sketch-based image retrieval (SBIR) can be efficiently tackled by cross-modal binary representation learning methods, where Hamming distance matching significantly speeds up the process of similarity search. Providing training and test data subjected to a fixed set of pre-defined categories, the cutting-edge SBIR and cross-modal hashing works obtain acceptable retrieval performance. However, most of the existing methods fail when the categories of query sketches have never been seen during training. In this paper, the above problem is briefed as a novel but realistic zero-shot SBIR hashing task. We elaborate the challenges of this special task and accordingly propose a zero-shot sketch-image hashing (ZSIH) model. An end-to-end three-network architecture is built, two of which are treated as the binary encoders. The third network mitigates the sketch-image heterogeneity and enhances the semantic relations among data by utilizing the Kronecker fusion layer and graph convolution, respectively. As an important part of ZSIH, we formulate a generative hashing scheme in reconstructing semantic knowledge representations for zero-shot retrieval. To the best of our knowledge, ZSIH is the first zero-shot hashing work suitable for SBIR and cross-modal search. Comprehensive experiments are conducted on two extended datasets, i.e., Sketchy and TU-Berlin with a novel zero-shot train-test split. The proposed model remarkably outperforms related works.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “Contemporary research on this category is also directed towards zero-shot SBIR [12, 67, 44] and binary hash-code embedding [30, 49].”
A Survey of Zero-Shot Learning (2019, background): “In [55, 133, 167], zero-shot learning is combined with hashing and forms zero-shot hashing problems that aim to hash images of unseen classes.”
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “We compare our model with three prior works on ZSSBIR: ZSIH [35], CAAE and CVAE [16], and SEMPCYC [4], which all use generative models and complicated frameworks, e.”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Further efforts have been made through zero-shot SBIR [13, 56] for cross-category generalisation, and employing binary hash-code embedding [29, 43] to reduce the computational complexity.”
Generalising Fine-Grained Sketch-Based Image Retrieval (2019, methodology): “focused on a three branch hashing network for efficient SBIR [27].”
Deep Learning for Free-Hand Sketch: A Survey (2020, unknown): “photo hashing to the zero-shot setting [261]; and training SBIR”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “tion [29] to improve generalisability, and alleviating sketchimage heterogeneity via Kronecker fusion layer with graph convolution [50], thus enhancing semantic relations among data towards a generative hashing scheme for ZS-SBIR.”
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “ZSIH [28] did not report the specific details on their split (other than 25 classes were used for testing), and we could not produce their results on QuickDraw-Extended due to the lack of publicly available code.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “Our results in Table 1 (last two lines) show that our model significantly outperforms [43], due to the benefit of our cross-modal adversarial mechanism and heterogeneous side information.”
Sketch-Guided Object Localization in Natural Images (2020, background): “Despite the problem of sketch-guided object localization has a relation to the well-studied sketch-based image retrieval (SBIR) problem [5,14,20,25,28,30,38,40], these are distinctively different in terms of objectives.”
A Meta-Learning Framework for Generalized Zero-Shot Learning (2019, methodology): “Because of the wide applicability and more realistic setting the ZSL framework also applied on the different domain like Zero-Shot Task Transfer [34], zero-shot sketch-based image retrieval [42, 23], zero-shot knowledge distillation [31], zero-shot action recognition [56, 13, 30, 28] etc.”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Category-level SBIR: Category-level sketch-photo retrieval is now well studied [3, 43, 2, 6, 5, 47, 39, 9, 24, 4, 23].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, background): “Other related problems such as zero-shot SBIR [9, 10, 24] and sketch-photo hashing [23, 35] are also studied.”