Enhancing Sketch-Based Image Retrieval by CNN Semantic Re-ranking

By Luo Wang, Xueming Qian, Yuting Zhang, Jialie Shen, Xiaochun Cao. 2020

117 citations recorded by the source corpus.

This paper introduces a convolutional neural network (CNN) semantic re-ranking system to enhance the performance of sketch-based image retrieval (SBIR). Distinguished from the existing approaches, the proposed system can leverage category information brought by CNNs to support effective similarity measurement between the images. To achieve effective classification of query sketches and high-quality initial retrieval results, one CNN model is trained for classification of sketches, another for that of natural images. Through training dual CNN models, the semantic information of both the sketches and natural images is captured by deep learning. In order to measure the category similarity between images, a category similarity measurement method is proposed. Category information is then used for re-ranking. Re-ranking operation first infers the retrieval category of the query sketch and then uses the category similarity measurement to measure the category similarity between the query sketch and each initial retrieval result. Finally, the initial retrieval results are re-ranked. The experiments on different types of SBIR datasets demonstrate the effectiveness of the proposed re-ranking method. Comparisons with other re-ranking algorithms are also given to show the proposed method’s superiority. Further, compared to the baseline systems, the proposed re-ranking approach achieves significantly higher precision in the top ten different SBIR methods and datasets.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Shiv Ram Dubey
2. Xianglong Liu
3. Dacheng Tao
4. Shihao Bai
5. Aishan Liu
6. Edwin R. Hancock
7. Yuqing Ma
8. Ling Shao

In Representation & Self-Supervised Learning

1. Dacheng Tao
2. Shiv Ram Dubey
3. Xianglong Liu

Citations in the corpus

• Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, background): “Notable successes include image classification [27]–[29], image generation [30]–[32], representation learning [33], [34], image retrieval [35], [36], object detection [37]–[39], video applications [40], and image translation [41].”
• A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, methodology): “different modes, such as unsupervised image retrieval [112], [114], [118], [113], supervised image retrieval [95], [97], [103], [106], semi-supervised image retrieval [141], crossmodal retrieval [134], [142], sketch based image retrieval [151], [158], and object retrieval [126], [165].”