Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval

By Anjan Dutta, Zeynep Akata. 2019

162 citations recorded by the source corpus.

Zero-shot sketch-based image retrieval (SBIR) is an emerging task in computer vision, allowing to retrieve natural images relevant to sketch queries that might not been seen in the training phase. Existing works either require aligned sketch-image pairs or inefficient memory fusion layer for mapping the visual information to a semantic space. In this work, we propose a semantically aligned paired cycle-consistent generative (SEM-PCYC) model for zero-shot SBIR, where each branch maps the visual information to a common semantic space via an adversarial training. Each of these branches maintains a cycle consistency that only requires supervision at category levels, and avoids the need of highly-priced aligned sketch-image pairs. A classification criteria on the generators' outputs ensures the visual to semantic space mapping to be discriminating. Furthermore, we propose to combine textual and hierarchical side information via a feature selection auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in zero-shot SBIR performance over the state-of-the-art on the challenging Sketchy and TU-Berlin datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “Existing zero-shot SBIR [12, 14] works split the dataset into 104 / 21 disjoint classes for training/testing(unseen).”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “Current directions include extending SBIR to the generalized zero-shot setting, where testing categories are a mix of training and unseen categories [245], [251]; extending sketch-”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Whereas zero-shot SBIR [47, 6, 9, 24] asks models to generalise across disjoint training and testing classes in order to alleviate annotation costs.”
Learning Robust Representations via Multi-View Information Bottleneck (2020, methodology): “Following recent work (Zhang et al., 2018; Dutta & Akata, 2019), we use features extracted from images and sketches by a VGG (Simonyan & Zisserman, 2014) architecture trained for classiﬁcation “cat” “apple” Method mAP@all Prec@200 SaN (Yu et al., 2017) 0.208 0.292 GN Triplet (Sangkloy et al., 2016)…”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, background): “Research has flourished in recent years, where the main focus has been on addressing the sketch-photo domain gap [33, 16, 44] and data scarcity [5, 3, 10, 35, 12].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “ZS-Cross [65] with its simplistic cross-modal training paradigm is quickly surpassed over by ZS-CCGAN [16] (by 0.116 mAP@all on Sketchy), as the latter is aided with a cycle consistency loss in an adversarial training paradigm in addition to the guidance from word2vec embeddings of categories – prov”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, methodology): “The area is split into ﬁne-grained (FG-SBIR) (Yu et al., 2016; Sangkloy et al., 2016; Bhunia et al., 2020) and a zero-shot setting (ZS-SBIR) (Dutta & Akata, 2019; Pandey et al., 2020; Dey et al., 2019).”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Thanks to the convenience of interactive touchscreen devices, sketch-based image retrieval (SBIR) [12, 13, 15, 39] has emerged as a practical means of image research that is complementary to the conventional text-based retrieval [26].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, background): “Other related problems such as zero-shot SBIR [9, 10, 24] and sketch-photo hashing [23, 35] are also studied.”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, background): “Sketch Representation Learning: Learning good sketch representations benefits a variety of sketch-specific problems like classification [64], retrieval [62], scene understanding [38], sketch based image retrieval [7, 45, 46, 14, 16, 4, 53, 52], generative sketch modelling [57, 22, 6] etc.”
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “We compare our model with three prior works on ZSSBIR: ZSIH [35], CAAE and CVAE [16], and SEMPCYC [4], which all use generative models and complicated frameworks, e.”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, background): “developed by Dutta and Akata (2019) for zero-shot SBIR [209].”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “With the ever rising popularity of touch screen devices, sketch-based image retrieval (SBIR) has witnessed significant interest within the vision community [5, 34, 35, 50, 11, 45].”