A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts

By Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, Ahmed M. Elgammal. 2017

402 citations recorded by the source corpus.

Most existing zero-shot learning methods consider the problem as a visual semantic embedding one. Given the demonstrated capability of Generative Adversarial Networks(GANs) to generate images, we instead leverage GANs to imagine unseen categories from text descriptions and hence recognize novel classes with no examples being seen. Specifically, we propose a simple yet effective generative model that takes as input noisy text descriptions about an unseen class (e.g. Wikipedia articles) and generates synthesized visual features for this class. With added pseudo data, zero-shot learning is naturally converted to a traditional classification problem. Additionally, to preserve the inter-class discrimination of the generated features, a visual pivot regularization is proposed as an explicit supervision. Unlike previous methods using complex engineered regularizers, our approach can suppress the noise well without additional regularization. Empirically, we show that our method consistently outperforms the state of the art on the largest available benchmarks on Text-based Zero-shot Learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Epidemiology, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Epidemiology

In Natural Language Processing

In Generative Models

Citations in the corpus

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “It also drives research progress in multimodal learning and inference across vision and language, which is one of the most active research areas in recent years [20, 18, 36, 19, 41, 4, 30, 5, 1, 31, 33, 32]”
Rectification-based Knowledge Retention for Continual Learning (2021, background): “Another popular approach is to generate images/features of unseen classes by using their class embeddings [7, 41].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “GANs have also been used in other machine learning tasks [290] such as active learning [291], [292], online learning [293], ensemble learning [294], zero-shot learning [295], [296], and multi-task learning [297].”
Spot and Learn: A Maximum-Entropy Patch Sampler for Few-Shot Image Classification (2019, background): “Another branch of algorithms focus on data hallucination to generate more training samples [9, 29, 32].”
Leveraging the Invariant Side of Generative Zero-Shot Learning (2019, methodology): “Following previous work [35, 40], we deploy the PASCAL VOC 2008 as seen dataset and the Yahoo as unseen one.”
A Meta-Learning Framework for Generalized Zero-Shot Learning (2019, methodology): “However, unlike recent works on synthesis based ZSL models [26, 63, 46, 53, 10], we endow the generator the capability to meta-learn using very few examples per seen class.”
Adaptive Cross-Modal Few-Shot Learning (2019, neutral): “ to embed information for both modalities and align the distribution of the two latent spaces. Data synthetic methods rely on generative models to generate image or image feature as data augmentation [60, 57, 31, 54] for unseen data to train the mapping function for more robust alignment. ZSL does n”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, background): “Convolutional Neural Networks (CNNs) have successfully tackled classic computer vision problems such as image classification [12, 29, 31, 52], object detection [19, 46, 55, 63, 74, 79] and generation [43, 58, 71, 73, 80], where the input image has a grid-like structure.”
A Review of Generalized Zero-Shot Learning Methods (2020, methodology): “In this regards, GZSL-based frameworks have been developed for different NLP applications such as text classification with single label [90], [162], multi-label [102], [233] as well as noisy text description [46], [47].”
Region Graph Embedding Network for Zero-Shot Learning (2020, background): “Moreover, to address the domain bias issue (Fig.1(a)-(b)) in the end-to-end GZSL models [52, 80, 79], we propose a balance loss by minimizing the maximum response consistency between seen and unseen predictions.”
Shifting More Attention to Video Salient Object Detection (2019, background): “More recently, deep learning based VSOD models [31, 38, 39, 41, 67, 68, 76] have gained more attention inspired by the success of applying deep neural networks on image saliency detection [13–15,32,50,71,72,86,88–90,94].”
Don’t Even Look Once: Synthesizing Features for Zero-Shot Detection (2019, background): “Motivated by these concerns, recent works have proposed to synthesize unseen examples by means of generative models such as autoencoders [4, 19], GANs and adversarial methods [49, 20, 42, 16, 23, 36] that take semantic vectors as input, and output images.”
Classifier and Exemplar Synthesis for Zero-Shot Learning (2018, methodology): “Once those examples are generated, one can employ any supervised learning technique to learn classifiers [36,90,77,7].”
Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation (2018, background): “Generative Adversarial Networks (GANs) [12, 45, 47] includes two networks: generator and discriminator which compete against each other to improve performances.”
Attribute Prototype Network for Zero-Shot Learning (2020, background): “Recent advances in zero-shot learning mainly focus on learning better visual-semantic embeddings [26, 50, 17, 5] or training generative models to synthesize features [44, 45, 57, 56, 19, 33].”
Camouflaged Object Detection (2020, background): “New techniques such as weakly supervised learning [53, 54], zero-shot learning [84], VAE [85], and multiscale backbone [20] could also be explored.”