F-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning

By Yongqin Xian, Saurabh Sharma, Bernt Schiele, Zeynep Akata. 2019

571 citations recorded by the source corpus.

When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zero- and few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Generative Feature Replay For Class-Incremental Learning (2020, background): “The same idea is further improved in [52], where they combine a better feature generator by combining the strength of a VAE and a GAN.”
Meta-Transfer Learning Through Hard Tasks (2019, methodology): “ditioned on Gaussian noises [11]–[13] or object attributes [14].”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, background): “rformance on all benchmarks. 2 Related Works Few-shot learning & meta-learning.Research literature on few-shot learning paradigms exhibits a high diversity from using data augmentation techniques [9, 75,77] over sharing feature representation [2,76] to meta-learning [18,72]. In this paper, we fo”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, methodology): “While some methods propose to hallucinate additional training examples for the data-starved novel classes [16], [17], [18], [19], recent work is more focused on meta-learning [5], [20], [21], [22], [23], [24], [25], [26] , which we detail below.”
Context-aware Feature Generation For Zero-shot Semantic Segmentation (2020, methodology): “Many ZSL methods [1, 8, 9, 11, 27, 32, 33, 37, 46] attempted to learn a mapping between feature space and semantic embedding space.”
Any-Shot Object Detection (2020, background): “The third body of work addresses both zero and few-shot learning tasks together [32, 41, 26, 36].”
Contrastive Embedding for Generalized Zero-Shot Learning (2021, methodology): “To further mitigate the data imbalance problem, feature generation methods learn to complement the visual samples for unseen classes [7, 50, 38, 70, 72, 56, 59, 64].”
Synthesized Feature based Few-Shot Class-Incremental Learning on a Mixture of Subspaces (2021, methodology): “Learning without forgetting using word vectors: Word vectors have shown promising success on various computer vision tasks such as zero-shot learning, few-shot learning, image/video captioning and visual question answering [15, 29, 39, 43, 9, 46, 47, 6, 5, 7].”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “On the other hand, few-shot learning (FSL) refers to the task of recognizing images or detecting objects with a model trained on very few samples [74,57].”
Bookworm continual learning: beyond zero-shot learning and continual learning (2020, background): “Recent works also use feature generators to synthesize features of unseen classes (Xian et al., 2018b; Mishra et al., 2018; Xian et al., 2019).”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “On the other hand, VAEGAN-D2 [182] learns the marginal feature distribution of the unlabeled samples using”
A Meta-Learning Framework for Generalized Zero-Shot Learning (2019, background): “Among these, [46, 4, 54] are based on the VAE architectures while [53, 7, 26, 10] use adversarial sample generation based on the class conditioned attribute.”
Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, background): “Some recent work investigates the connection between recognition and view synthesis, and makes some attempt to combine them together (Sun et al., 2018; Wang et al., 2018; Xian et al., 2019; Santurkar et al., 2019; Xiong et al., 2020; Michalkiewicz et al., 2020).”
Generalized Many-Way Few-Shot Video Classification (2020, background): “The evaluation is often performed only on novel classes, except [15,45,32] who evaluate on the combination of base+novel classes.”
Transductive Zero-Shot Learning for 3D Point Cloud Classification (2019, background): “[59] proposed f-VAEGAN-D2 which takes advantage of both VAEs and GANs to learn the feature distribution of unlabeled data.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “When re-formulated as a semi-supervised representation learner, GANs can leverage the benefits of both unsupervised generative modeling and supervised discriminative learning [66], [67], [68], [69], [113], [114], [115], [116], [117], [118].”