Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly

By Yongqin Xian, Christoph H. Lampert, Bernt Schiele, Zeynep Akata. 2017

1,854 citations recorded by the source corpus.

Due to the importance of zero-shot learning, i.e., classifying images where there is a lack of labeled training data, the number of proposed approaches has recently increased steadily. We argue that it is time to take a step back and to analyze the status quo of the area. The purpose of this paper is three-fold. First, given the fact that there is no agreed upon zero-shot learning benchmark, we first define a new benchmark by unifying both the evaluation protocols and data splits of publicly available datasets used for this task. This is an important contribution as published results are often not comparable and sometimes even flawed due to, e.g., pre-training on zero-shot test classes. Moreover, we propose a new zero-shot learning dataset, the Animals with Attributes 2 (AWA2) dataset which we make publicly available both in terms of image features and the images themselves. Second, we compare and analyze a significant number of the state-of-the-art methods in depth, both in the classic zero-shot setting but also in the more realistic generalized zero-shot setting. Finally, we discuss in detail the limitations of the current status of the area which can be taken as a basis for advancing it.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

Citations in the corpus

Learning Visual Representations with Caption Annotations (2020, background): “Using the recently proposed splits [19], we have 150 (resp.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “We follow one of the influential related surveys (Xian et al., 2018), which says “At test time, in zero-shot learning setting, the aim is to assign a test image to an unseen class label” where involving unseen class labels is a key.”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, neutral): “Zero-shot learning [35,36,65] is an alternative approach aiming to develop models that can handle a new task without task-specific data or adaptation protocols.”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “Openset recognition allows new concepts in evaluation, while typically remains the same visual domain [87, 93]; Domain shift allows new visual domain in evaluation, while typically remains the same concept pool [63, 28].”
Learning Open-World Object Proposals without Learning to Classify (2021, methodology): “This line of research is also related to few-shot [33] and zeroshot detection[34], [35] methods, which attempt to detect novel classes given only a few samples or class descriptions.”
Conditional variance penalties and domain shift robustness (2017, methodology): “We work with the ‘Animals with attributes 2’ (AwA2) dataset (Xian et al., 2017) and consider classifying images of horses and elephants.”
Learning to Compare: Relation Network for Few-Shot Learning (2017, methodology): “Results under the GBU setting We follow the evaluation setting of [42].”
Rectification-based Knowledge Retention for Continual Learning (2021, background): “We experiment with four datasets for the task incremental generalized zero-shot learning (GZSL) problem i.e. Attribute Pascal and Yahoo (aPY) [6], Animals with Attributes 1 (AWA1) [31], Caltech-UCSD-Birds 200-2011 (CUB) [29], and SUN Attribute dataset (SUN) [19].”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, methodology): “Nearest Class Mean (NCM) Classifier The NCM classifier and its variants have been widely used in few-shot or zero-shot learning [56, 59].”
Efficient Lifelong Learning with A-GEM (2018, background): “, 2017), and it is also a key ingredient in all the zero/few-shot learning algorithms (Lampert et al., 2014; Xian et al., 2018; Elhoseiny et al., 2017; Wah et al., 2011; Lampert et al., 2009).”
A survey on Image Data Augmentation for Deep Learning (2019, background): “• One-shot and Zero-shot learning [16, 17] algorithms represent another paradigm for building models with extremely limited data.”
Zero-Shot Object Detection (2018, background): “17% [53]) at IoU ≥ 0 .”
Generalizing from a Few Examples (2019, background): “ot learning [68] recognizes a new class with no supervised training examples by linking them to existing classes, which usually count on external data sources such as text corpus and lexical database [135]. FSL does not need to be a transfer learning problem. However, when the given supervised infor”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “The idea of embedding images and text into a shared space has been used to achieve “zero-shot” generalization for a long time [10,36,40].”
Recent Advances in Open Set Recognition: A Survey (2018, background): “Therefore, some researchers have begun to pay attention to the more generalized ZSL (GZSL) [30], [31], [32], [33], where the testing samples come from both KKCs and UKCs.”
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders (2018, methodology): “All hyperparameters were chosen on a validation set provided by [35].”
SemGIF: A Semantics Guided Incremental Few-shot Learning Framework with Generative Replay (2021, positive): “For the incremental stage where evaluation over the base and novel dataset is considered, we report the harmonic mean as defined in [42].”
Attribute Prototype Network for Zero-Shot Learning (2020, methodology): “The goal for ZSL is to predict the label of images from unseen classes, i.e., X → Y u , while for generalized ZSL (GZSL) [42] the goal is to predict images from both seen and unseen classes, i.e., X → Y u ∪ Y s .”