Sketch3T: Test-Time Training for Zero-Shot SBIR

By Aneeshan Sain, Ayan Kumar Bhunia, Vaishnav Potlapalli, Pinaki Nath Chowdhury, Tao A. Xiang, Yi-Zhe Song. 2022

71 citations recorded by the source corpus.

Zero-shot sketch-based image retrieval typically asks for a trained model to be applied as is to unseen categories. In this paper, we question to argue that this setup by definition is not compatible with the inherent abstract and subjective nature of sketches – the model might transfer well to new categories, but will not understand sketches existing in different test-time distribution as a result. We thus extend ZS-SBIR asking it to transfer to both categories and sketch distributions. Our key contribution is a test-time training paradigm that can adapt using just one sketch. Since there is no paired photo, we make use of a sketch raster-vector reconstruction module as a self-supervised auxiliary task. To maintain the fidelity of the trained cross-modal joint embedding during test-time update, we design a novel meta-learning based training paradigm to learn a separation between model updates incurred by this auxiliary task from those off the primary objective of discriminative learning. Extensive experiments show our model to outperform state-of-the-arts, thanks to the proposed test-time adaption that not only transfers to new categories but also accommodates to new sketching styles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “Contemporary research on this category is also directed towards zero-shot SBIR [12, 67, 44] and binary hash-code embedding [30, 49].”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Related Work Fine-grained sketch based image retrieval The ability of sketches to offer inherently fine-grained visual descriptions commences the avenues of fine-grained sketch-based image retrieval (FG-SBIR) [4, 7, 8, 53, 54, 58, 63].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, background): “Thanks to the convenience of interactive touchscreen devices, sketch-based image retrieval (SBIR) [12, 13, 15, 39] has emerged as a practical means of image research that is complementary to the conventional text-based retrieval [26].”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “, person re-identification [457], [582] and sketch-to-image retrieval [433], [445].”