Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval
By Ayan Kumar Bhunia, Subhadeep Koley, Abdullah Faiz Ur Rahman Khilji, Aneeshan Sain, Pinaki Nath Chowdhury, Tao A. Xiang, Yi-Zhe Song. 2022
58 citations recorded by the source corpus.
Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., “I can't sketch”) has however proven to be fatal for its widespread adoption. This paper tackles this “fear” head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the “I can't sketch”. We consequently design a stroke subset selector that detects noisy strokes, leaving only those which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Aneeshan Sain
- Ayan Kumar Bhunia
- Tao A. Xiang
- Yi-Zhe Song
- Pinaki Nath Chowdhury
- Viswanatha Reddy Gajjala
- Vaishnav Potlapalli
- Yongxin Yang
Citations in the corpus
- Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, neutral): “Sketch Based Image Retrieval (SBIR): SBIR aims at retrieving paired photo given a query sketch, either at a category-level [11, 67, 12, 41, 65] or at a finer-grained instance level (FG-SBIR) [46, 10, 5, 6, 2].”
- Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Research thus advanced to finegrained SBIR [7, 38, 52] beginning with deformable-part models [25].”
- Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Related Work Fine-grained sketch based image retrieval The ability of sketches to offer inherently fine-grained visual descriptions commences the avenues of fine-grained sketch-based image retrieval (FG-SBIR) [4, 7, 8, 53, 54, 58, 63].”