Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval

By Qing Liu, Lingxi Xie, Huiyu Wang, Alan L. Yuille. 2019

129 citations recorded by the source corpus.

Sketch-based image retrieval (SBIR) is widely recognized as an important vision problem which implies a wide range of real-world applications. Recently, research interests arise in solving this problem under the more realistic and challenging setting of zero-shot learning. In this paper, we investigate this problem from the viewpoint of domain adaptation which we show is critical in improving feature embedding in the zero-shot scenario. Based on a framework which starts with a pre-trained model on ImageNet and fine-tunes it on the training set of SBIR benchmark, we advocate the importance of preserving previously acquired knowledge, e.g., the rich discriminative features learned from ImageNet, to improve the model's transfer ability. For this purpose, we design an approach named Semantic-Aware Knowledge prEservation (SAKE), which fine-tunes the pre-trained model in an economical way and leverages semantic information, e.g., inter-class relationship, to achieve the goal of knowledge preservation. Zero-shot experiments on two extended SBIR datasets, TU-Berlin and Sketchy, verify the superior performance of our approach. Extensive diagnostic experiments validate that knowledge preserved benefits SBIR in zero-shot settings, as a large fraction of the performance gain is from the more properly structured feature embedding for photo images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al. 2019c).”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, background): “Other related problems such as zero-shot SBIR [9, 10, 24] and sketch-photo hashing [23, 35] are also studied.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “ learning a meta-learner that can automatically learn what knowledge to transfer from the source network to where in the target network. Prior knowledge about data: Dvornik et al. [45] and Liu et al. [125] address the third type of prior knowledge, namely data variance. To be speciﬁc, in [45], en en”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Whereas zero-shot SBIR [47, 6, 9, 24] asks models to generalise across disjoint training and testing classes in order to alleviate annotation costs.”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “ZS-SAKE [29] employs knowledgedistillation paradigm using teacher signal from an ImageNet pre-trained CNN model and constrained by semantic information from category-labels to retrieve in a Zero-shot setting.”
Deep Learning for Free-Hand Sketch: A Survey (2020, unknown): “[245], [246], [247], [248], [249], [250], [251], [252], [253], [254], [255].”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Learning a compatibility or a matching function between multiple modalities in zero-shot scenario [58,11,37] requires structure in the class embedding space where the image features are mapped to.”