Multi-modal Cycle-consistent Generalized Zero-Shot Learning

By Rafael Felix, B. V. Kumar, Ian David Reid, Gustavo Carneiro. 2018

386 citations recorded by the source corpus.

In generalized zero shot learning (GZSL), the set of classes are split into seen and unseen classes, where training relies on the semantic features of the seen and unseen classes and the visual representations of only the seen classes, while testing uses the visual representations of the seen and unseen classes. Current methods address GZSL by learning a transformation from the visual to the semantic space, exploring the assumption that the distribution of classes in the semantic and visual spaces is relatively similar. Such methods tend to transform unseen testing visual representations into one of the seen classes’ semantic features instead of the semantic features of the correct unseen class, resulting in low accuracy GZSL classification. Recently, generative adversarial networks (GAN) have been explored to synthesize visual representations of the unseen classes from their semantic features - the synthesized representations of the seen and unseen classes are then used to train the GZSL classifier. This approach has been shown to boost GZSL classification accuracy, but there is one important missing constraint: there is no guarantee that synthetic visual representations can generate back their semantic feature in a multi-modal cycle-consistent manner. This missing constraint can result in synthetic visual representations that do not represent well their semantic features, which means that the use of this constraint can improve GAN-based approaches. In this paper, we propose the use of such constraint based on a new regularization for the GAN training that forces the generated visual features to reconstruct their original semantic features. Once our model is trained with this multi-modal cycle-consistent semantic compatibility, we can then synthesize more representative visual representations for the seen and, more importantly, for the unseen classes. Our proposed approach shows the best GZSL classification results in the field in several publicly available datasets.

Subfields: Artificial Intelligence, Cancer Research, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Media Technology

Citations in the corpus

Rectification-based Knowledge Retention for Continual Learning (2021, methodology): “Another popular approach is to generate images/features of unseen classes by using their class embeddings [7, 41].”
Recent Advances in Open Set Recognition: A Survey (2018, background): “Therefore, some researchers have begun to pay attention to the more generalized ZSL (GZSL) [30], [31], [32], [33], where the testing samples come from both KKCs and UKCs.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “The ZSL methods that we evaluate are: CMT [45], DeViSE [15], SSE [66], JLSE [67], ZSH [56], SAE [24] and FRWGAN [14].”
Gradient Matching Generative Networks for Zero-Shot Learning (2019, methodology): “Towards addressing this problem, several recent work have proposed generative models that can synthesize training samples for unseen classes and learn a classiﬁer from real and/or synthesized examples [25, 26, 27, 28].”
F-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning (2019, methodology): “asses at test time, i.e. inductive zero-shot setting [29, 13], or the samples can come from all classes but then their labels are not known, i.e. transductive zero-shot setting [15, 48]. Recent works [58, 28, 11] address generalized zero-shot learning by generating synthetic CNN features of unseen c”
Context-aware Feature Generation For Zero-shot Semantic Segmentation (2020, methodology): “Inspired by feature generation methods for zero-shot classification [7, 44], ZS3Net learns to generate pixel-wise features from semantic word embeddings.”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “In addition, in studies [160], [161], [162], a multi-modal cycle consistency loss function is developed to preserve the semantic consistency of the generated visual features.”
Adaptive and Generative Zero-Shot Learning (2021, background): “…the dropping of semantic information by disentangling the semantic space into two subspaces, one for classiﬁcation and the other for reconstruction; Felix et al. (2018) enforces visual-semantic feature consistency by requiring synthesized visual features to reconstruct semantic features…”
Region Graph Embedding Network for Zero-Shot Learning (2020, neutral): “6 – – – – cycle-CLSWGAN(ECCV’18) [13] 58.”
ProtoGAN: Towards Few Shot Learning for Action Recognition (2019, methodology): “To ensure that the generated features contain class semantics for subsequent supervised classification, similar to [5], an decoder (h) is used.”
A Meta-Learning Framework for Generalized Zero-Shot Learning (2019, methodology): “This setting is popularly known as Zero-Shot Learning (ZSL), and has drawn a considerable interest recently [44, 32, 47, 5, 38, 53, 46, 27, 39, 7, 29, 4, 33, 10, 62, 41, 23].”
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort (2021, methodology): “For the purpose of zero-shot image classification, GANs have also been used for synthesizing visual features of unseen classes from their semantic features [5, 36, 14, 44].”
Lifelong Zero-Shot Learning (2020, methodology): “Besides, there are some ZSL methods [Felix et al. , 2018; Zhu et al. , 2018], which train generative models to obtain the features of unseen classes.”
Adversarial Fine-Grained Composition Learning for Unseen Attribute-Object Recognition (2019, methodology): “Generally, most of the GAN-based ZSL methods [3, 5, 26, 8] inherit the inspiration of cGAN [17] which extends GAN from unsupervised learning into the semi-supervised setting by inputting the conditional variables along with the noise vectors.”
Normalization Matters in Zero-Shot Learning (2020, neutral): “[17] Rafael Felix, Vijay B.”