Generalising Fine-Grained Sketch-Based Image Retrieval

By Kaiyue Pang, Ke Li, Yongxin Yang, Honggang Zhang, Timothy M. Hospedales, Tao A. Xiang, Yi-Zhe Song. 2019

97 citations recorded by the source corpus.

Fine-grained sketch-based image retrieval (FG-SBIR) addresses matching specific photo instance using free-hand sketch as a query modality. Existing models aim to learn an embedding space in which sketch and photo can be directly compared. While successful, they require instance-level pairing within each coarse-grained category as annotated training data. Since the learned embedding space is domain-specific, these models do not generalise well across categories. This limits the practical applicability of FG-SBIR. In this paper, we identify cross-category generalisation for FG-SBIR as a domain generalisation problem, and propose the first solution. Our key contribution is a novel unsupervised learning approach to model a universal manifold of prototypical visual sketch traits. This manifold can then be used to paramaterise the learning of a sketch/photo representation. Model adaptation to novel categories then becomes automatic via embedding the novel sketch in the manifold and updating the representation and retrieval function accordingly. Experiments on the two largest FG-SBIR datasets, Sketchy and QMUL-Shoe-V2, demonstrate the efficacy of our approach in enabling cross-category generalisation of FG-SBIR.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “We are also distinctly different to the parallel problem of SBIR – SBIR typically get exposed to paired sketch-photo data at training to learn a cross-modal embedding; we on the other hand need to work with sketches only at training (i.e., no photo information whatsoever), yet still aim to generate ”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Datasets: We use QMUL-Shoe-V2 [29, 34, 40] and QMUL-Chair-V2 [40] datasets that have been specifically designed for FG-SBIR.”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “In recent years, motivated by the zero-shot setting for supervised photo retrieval [241], zero-shot sketch-based image retrieval (ZSSBIR) has also been studied [65], [216], [242], [243], [244],”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Related Work Fine-grained sketch based image retrieval The ability of sketches to offer inherently fine-grained visual descriptions commences the avenues of fine-grained sketch-based image retrieval (FG-SBIR) [4, 7, 8, 52, 53, 57, 62].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Research thus advanced to finegrained SBIR [7, 38, 52] beginning with deformable-part models [25].”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Additionally, fine-grained SBIR [45,44] is an alternative sketchbased image retrieval task, allowing to search for specific object images, which has already received remarkable attention in the computer vision community.”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, methodology): “It adapts using inner loop updates across retrieval tasks over categories in SBIR and over instances in FG-SBIR frameworks”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, methodology): “11) and ﬁne-grained sketch-based image retrieval (FG-SBIR, cf. Fig.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, methodology): “Cross-category generalisation [26] and on-the-fly retrieval setup [5] are more recent additions to existing FG-SBIR literature.”
Sketchformer: Transformer-Based Representation for Sketched Structure (2020, background): “The generalization of sketch embeddings beyond training classes have also been studied [28, 29], and parameterized for zero-shot learning [30].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Initially starting with graph-matching models [33], FG-SBIR research gained traction with the advent of various deeplearning based approaches [56, 46, 7, 3].”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, background): “Sketch Representation Learning: Learning good sketch representations benefits a variety of sketch-specific problems like classification [64], retrieval [62], scene understanding [38], sketch based image retrieval [7, 45, 46, 14, 16, 4, 53, 52], generative sketch modelling [57, 22, 6] etc.”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “At its inception, fine-grained sketch-based image retrieval (FG-SBIR) focussed on retrieving photos of commercial objects [1, 30, 50].”