Fine-Grained Image Retrieval: the Text/Sketch Input Dilemma

By Jifei Song, Yi-Zhe Song, Tony Xiang, Timothy M. Hospedales. 2017

64 citations recorded by the source corpus.

Fine-grained image retrieval (FGIR) enables a user to search for a photo of an object instance based on a mental picture. Depending on how the object is described by the user, two general approaches exist: sketch-based FGIR or text-based FGIR, each of which has its own pros and cons. However, no attempt has been made to systematically investigate how informative each of these two input modalities is, and more importantly whether they are complementary to each thus should be modelled jointly. In this work, for the ﬁrst time we introduce a multi-modal FGIR dataset with both sketches and sentences description provided as query modalities. A multi-modal quadruplet deep network is formulated to jointly model the sketch and text input modalities as well as the photo output modality. We show that on its own the sketch modality is much more informative than text and each modality can beneﬁt the other when they are modelled jointly.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “We show that sketches are better suited as class support in comparison to text, thanks to its inherent ﬁne-grained nature, validated by ﬁndings in contemporary sketch literature [52, 11, 6].”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Majority of such models use variant of siamese networks [36, 42, 58, 46] that are suitable for cross-modal retrieval.”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, background): “Work in learning joint image-text spaces [3, 5, 10, 12, 20, 36] aims to project an image and a matching piece of text in such a way that the distance between related text and image is minimized.”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, methodology): “Unlike existing frameworks in FG-SBIR [64, 79] that independently map sketch and photo to a joint embedding sketch-photo space, Sain et al.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, methodology): “SN-HOLEF [38] is an extension over [48] employing spatial attention along with higher order ranking loss.”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, methodology): “This was further enhanced via attention based techniques with advanced higher order retrieval loss [49], hybrid generativediscriminative cross-domain image generation [36], textual tags [48] and employing mixed modal jigsaw solving for a better pre-training strategy [37].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Fine-grained SBIR: Sketch holds a noteworthy advantage in its potential to depict ﬁne-grained properties of the target image, which are hard to describe via other query mediums [46] like text or attribute.”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Attention mechanisms along with higher-order losses [54], hybrid generative-discriminative cross-domain image generation [39], textual tags [53] and mixed-modal jigsaw solving based pre-training strategy [40], enhanced it further.”
Deep Learning for Free-Hand Sketch: A Survey (2020, unknown): “Deep Sketch-based image retrieval (SBIR) has been widely studied [4], [7], [162], [163], [164], [165], [166], [167], [168], [169], [170], [171], [172], [173] in recent years.”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, background): “[183] showed the superiority of sketch over text for fine-grained retrieval, illustrating that each modality can complement the other when they are”
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval (2018, background): “Problems studied so far range from sketch recognition [3, 11, 33], sketch-based image retrieval (SBIR) [32, 20, 29, 31, 23, 30, 22, 10], to sketch synthesis [12].”