Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders

By Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, Zeynep Akata. 2018

693 citations recorded by the source corpus.

Many approaches in generalized zero-shot learning rely on cross-modal mapping between the image feature space and the class embedding space. As labeled images are expensive, one direction is to augment the dataset by generating either images or image features. However, the former misses fine-grained details and the latter requires learning a mapping associated with class embeddings. In this work, we take feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by modality-specific aligned variational autoencoders. This leaves us with the required discriminative information about the image and classes in the latent features, on which we train a softmax classifier. The key to our approach is that we align the distributions learned from images and from side-information to construct latent features that contain the essential multi-modal information associated with unseen classes. We evaluate our learned latent features on several benchmark datasets, i.e. CUB, SUN, AWA1 and AWA2, and establish a new state of the art on generalized zero-shot as well as on few-shot learning. Moreover, our results on ImageNet with various zero-shot splits show that our latent features generalize well in large-scale settings.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, background): “proposed framework primarily with respect to the harmonic mean (hm/J ) [39, 40, 24] that is computed between base and novel performance in the joint space.”
Rectification-based Knowledge Retention for Continual Learning (2021, methodology): “Therefore, CCLL cannot be applied to CADA-VAE.”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, methodology): “based classiﬁcation paradigms on both SEEN and UNSEEN categories to resolve the transductive learning problem, which requires recomputing the centroids for SEEN categories after model updates. Others [32, 45, 58] usually ignore the explicit relationship between SEEN and UNSEEN categories, and learn ”
Variational Adversarial Active Learning (2019, methodology): “Variational autoencoders: Autoencoders have long been used to effectively learn a feature space and representation [2, 42].”
Lifelong Zero-Shot Learning (2020, methodology): “Cross and Distribution Aligned VAE (CACD-VAE) [Schonfeld et al. , 2019] is selected as the base model, which trains VAEs [Kingma and Welling, 2013] to encode and decode features of visual and semantic embeddings respectively, and uses the learned latent features to train a ZSL classiﬁer.”
Adaptive and Generative Zero-Shot Learning (2021, background): “Previous generative models (Schonfeld et al., 2019; Xian et al., 2019) synthesize unseen samples by GAN/VAE or their variants.”
SemGIF: A Semantics Guided Incremental Few-shot Learning Framework with Generative Replay (2021, methodology): “This notion is again reinforced by the results shown in Table 1, wherein we show how the performance of the model degrades in the absence of semantic fusion.”
A Meta-Learning Framework for Generalized Zero-Shot Learning (2019, background): “Broadly, these models learn to generate/synthesize “artificial” examples from unseen classes [46, 10, 53, 29, 26, 41, 20], conditioning on their class attributes, and learn a classifier using these synthesized examples.”
Learning Classifier Synthesis for Generalized Few-Shot Learning (2019, background): “Previous approaches [9, 10, 18, 26, 37] often try to resolve the transductive learning setup of GFSL.”
Synthesized Feature based Few-Shot Class-Incremental Learning on a Mixture of Subspaces (2021, methodology): “In contrast, Schonfeld et al. [30] used a VAE for the same purpose.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “On the other hand, few-shot learning (FSL) refers to the task of recognizing images or detecting objects with a model trained on very few samples [74,57].”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “CADA-VAE [169] learns the shared cross-modal latent features of image and class attributes through a set of distribution alignment (DA) and cross alignment (CA) loss functions.”
Zero-Shot Object Detection: Joint Recognition and Localization of Novel Concepts (2020, positive): “(d) The fourth kind of approaches use synthesized features to improve ZSL and GZSL performance (Xian et al. 2018b; Schonfeld et al. 2019; Xian et al. 2019).”
Adaptive Cross-Modal Few-Shot Learning (2019, methodology): “In (Schönfeld et al., 2018), the authors propose a method that encodes information from both sides using two VAEs (Kingma & Welling, 2014) and adding regularization to align the two latent spaces.”
Generalized Continual Zero-Shot Learning (2020, methodology): “Generative models have recently shown promising results for both ZSL and GZSL setups using synthesized examples for unseen classes to train a supervised classifier [36, 8, 44, 13, 32, 50, 20, 47].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “Experiments have shown [8], [148], [149] that adaptive combinations of two or more modalities are much better than unimodal FSL.”