Partially Does It: Towards Scene-Level FG-SBIR with Partial Input

By Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Viswanatha Reddy Gajjala, Aneeshan Sain, Tao A. Xiang, Yi-Zhe Song. 2022

31 citations recorded by the source corpus.

We scrutinise an important observation plaguing scene-level sketch research - that a significant portion of scene sketches are “partial”. A quick pilot study reveals: (i) a scene sketch does not necessarily contain all objects in the corresponding photo, due to the subjective holistic interpretation of scenes, (ii) there exists significant empty (white) regions as a result of object-level abstraction, and as a result, (iii) existing scene-level fine-grained sketch-based image retrieval methods collapse as scene sketches become more partial. To solve this “partial” problem, we advocate for a simple set-based approach using optimal transport (OT) to model cross-modal region associativity in a partially-aware fashion. Importantly, we improve upon OT to further account for holistic partialness by comparing intra-modal adjacency matrices. Our proposed method is not only robust to partial scene-sketches but also yields state-of-the-art performance on existing datasets.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, neutral): “Sketch Based Image Retrieval (SBIR): SBIR aims at retrieving paired photo given a query sketch, either at a category-level [11, 67, 12, 41, 65] or at a ﬁner-grained instance level (FG-SBIR) [46, 10, 5, 6, 2].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Sketch as a query [10] however, prides in its ability to model fine-grained details.”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, background): “Despite great strides made [4, 34, 11], the fear-to-sketch has proven to be fatal for its omnipresence – a “I can’t sketch” reply is often the end of it.”