More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval

By Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Aneeshan Sain, Yongxin Yang, Tao A. Xiang, Yi-Zhe Song. 2021

70 citations recorded by the source corpus.

A fundamental challenge faced by existing Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) models is the data scarcity – model performances are largely bottlenecked by the lack of sketch-photo pairs. Whilst the number of photos can be easily scaled, each corresponding sketch still needs to be individually produced. In this paper, we aim to mitigate such an upper-bound on sketch data, and study whether unlabelled photos alone (of which they are many) can be cultivated for performance gain. In particular, we introduce a novel semi-supervised framework for cross-modal retrieval that can additionally leverage large-scale unlabelled photos to account for data scarcity. At the center of our semi-supervision design is a sequential photo-to-sketch generation model that aims to generate paired sketches for unlabelled photos. Importantly, we further introduce a discriminator-guided mechanism to guide against unfaithful generation, together with a distillation loss-based regularizer to provide tolerance against noisy training samples. Last but not least, we treat generation and retrieval as two conjugate problems, where a joint learning procedure is devised for each module to mutually benefit from each other. Extensive experiments show that our semi-supervised model yields a significant performance boost over the state-of-the-art supervised alternatives, as well as existing methods that can exploit unlabelled photos for FG-SBIR.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “The advocate of sketches is largely motivated by the line of work examining human-centric characteristics of sketches in many parallel applications – notably image re-a trieval [12], where the ﬁne-grained nature of sketches is used to successfully conduct instance-level retrieval [2, 6, 46, 14, 11].”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, background): “Sketch Representation Learning: Learning good sketch representations benefits a variety of sketch-specific problems like classification [64], retrieval [62], scene understanding [38], sketch based image retrieval [7, 45, 46, 14, 16, 4, 53, 52], generative sketch modelling [57, 22, 6] etc.”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, background): “Research has flourished in recent years, where the main focus has been on addressing the sketch-photo domain gap [33, 16, 44] and data scarcity [5, 3, 10, 35, 12].”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, methodology): “Related Work Fine-grained sketch based image retrieval The ability of sketches to offer inherently fine-grained visual descriptions commences the avenues of fine-grained sketch-based image retrieval (FG-SBIR) [4, 7, 8, 52, 53, 57, 62].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “31% without any complicated joint-training of Semi-Sup [4], costly hierarchical paired embedding of Cross-Hier [40], or meta-learning cumbersome feature transformation layer of StyleMeUp [41].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Although further works have addressed low-resource data via semi-supervised learning [4], or style-diversity in sketches via meta-learning disentanglement [47], training during inference to bridge the train-test data distribution gap, remains unseen in SBIR.”