StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval

By Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao A. Xiang, Yi-Zhe Song. 2021

109 citations recorded by the source corpus.

Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An effective SBIR model needs to explicitly account for this style diversity, crucially, to generalise to unseen user styles. To this end, a novel style-agnostic SBIR model is proposed. Different from existing models, a cross-modal variational autoencoder (VAE) is employed to explicitly disentangle each sketch into a semantic content part shared with the corresponding photo, and a style part unique to the sketcher. Importantly, to make our model dynamically adaptable to any unseen user styles, we propose to meta-train our cross-modal VAE by adding two style-adaptive components: a set of feature transformation layers to its encoder and a regulariser to the disentangled semantic content latent code. With this meta-learning framework, our model can not only disentangle the cross-modal shared semantic content for SBIR, but can adapt the disentanglement to any unseen user style as well, making the SBIR model truly style-agnostic. Extensive experiments show that our style-agnostic model yields state-of-the-art performance for both category-level and instance-level SBIR.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “…[53], cross-domain translation [35], re-inforcement learning based on-the-ﬂy retrieval [6], semi-supervised retrieval [2], style-agnostic retrieval [46], etc. Sketch for Vision Tasks: Hand-drawn sketches, by nature, are enriched with various human visual system-like understanding abilities and are…”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “With the ever rising popularity of touch screen devices, sketch-based image retrieval (SBIR) has witnessed significant interest within the vision community [5, 34, 35, 50, 11, 45].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “Specifically, we use the popular model-agnostic meta-learning (MAML) algorithm [18] (enhanced to MAML++ [1]), due its compatibility with any model trained via gradient descent, and diverse application range with several variants [1,35,43,48].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, background): “31% without any complicated joint-training of Semi-Sup [4], costly hierarchical paired embedding of Cross-Hier [40], or meta-learning cumbersome feature transformation layer of StyleMeUp [41].”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, background): “Sketch Representation Learning: Learning good sketch representations benefits a variety of sketch-specific problems like classification [64], retrieval [62], scene understanding [38], sketch based image retrieval [7, 45, 46, 14, 16, 4, 53, 52], generative sketch modelling [57, 22, 6] etc.”