Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning

By Soravit Changpinyo, Wei-Lun Chao, Fei Sha. 2016

181 citations recorded by the source corpus.

Leveraging class semantic descriptions and examples of known objects, zero-shot learning makes it possible to train a recognition model for an object class whose examples are not available. In this paper, we propose a novel zero-shot learning model that takes advantage of clustering structures in the semantic embedding space. The key idea is to impose the structural constraint that semantic representations must be predictive of the locations of their corresponding visual exemplars. To this end, this reduces to training multiple kernel-based regressors from semantic representation-exemplar pairs from labeled data of the seen object categories. Despite its simplicity, our approach significantly outperforms existing zero-shot learning methods on standard benchmark datasets, including the ImageNet dataset with more than 20,000 unseen categories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Experience Grounds Language (2020, background): “These representations allow for pixel level masks and skeletal modeling, and can be extended to zero-shot settings targeting all 20,000 ImageNet categories (Chao et al., 2016; Changpinyo et al., 2017).”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, methodology): “[36] proposes a simple model by projecting class semantic representations into the visual feature space and performing nearest neighbor classifiers among those projected representations.”
Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions (2018, background): “Another line of approach has focused on learning generalizable instance embeddings [1, 4, 5, 13, 17, 26, 37, 40, 43] and uses those embeddings on simple classifiers such as nearest neighbor rules.”
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection (2022, background): “Under this setting, various works [5] try to find relationships between existing categories and unknown categories through pre-trained semantic/text features [46, 37, 40, 4, 13] knowledge graphs [43, 19, 51, 49] and so on.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Many other ZSL approaches learn nonlinear multi-modal embedding [45, 2, 53, 6, 64], where most of the methods focus to learn a non-linear mapping from the image space to the semantic space.”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “In contrast, the studies in [66], [94], [186] attempt to synthesize classifiers instead of generating visual features”
Few-Shot Learning With Global Class Representations (2019, methodology): “Inspired by generalized ZSL [4, 5, 14, 24, 6], we define three evaluation metrics for generalized FSL : 1) accb – the accuracy of classifying the data samples from the base classes to all the classes (both base and novel); 2) accun – the accuracy of classifying the data samples from the novel classe”
Small Sample Learning in Big Data Era (2018, background): “After that, Changpinyo et al. (2017) learned a mapping function such that the semantic representation of class can predict well its class exemplar (center) that characterized the clustering structure, and then the function was used to construct nearest-neighbor style classifiers.”
Rethinking Knowledge Graph Propagation for Zero-Shot Learning (2018, methodology): “Results indicated with ∗, †, and ‡ are taken from [4], [5], and [33], respectively.”
Learning Classifier Synthesis for Generalized Few-Shot Learning (2019, unknown): “deﬁnitions of objects, ZSL transfer the relationship between images and attributes learned from SEEN classes to UNSEEN ones, so as to recognize a novel instance with only its category-wise attributes [4, 5]. Generalized ZSL [7] extends this by calibrating a prediction bias to jointly predict between”
Multi-label Zero-Shot Learning with Structured Knowledge Graphs (2017, background): “suming label priors, while label-embedding based methods [3, 43, 6, 5, 4] project both input images and their labels onto a latent space to exploit label correlation.”
Generalized Zero-Shot Learning with Deep Calibration Network (2018, methodology): “The work [9] further imposed the structural constraint that semantic representations must be predictive of the locations of their corresponding visual exemplars.”
Domain-Invariant Projection Learning for Zero-Shot Recognition (2018, unknown): “2.5j– – SJE [1] G N 73.9 51.7 – – j56.1 – JLSE [68] V N 80.5 42.1 50.4 83.8j– – SynC [8] G N 72.9 54.7 – – j62.7 – SAE [24] G N 84.7 61.4 55.4 91.5j65.2 27.2 LAD [21] V N 82.5 56.6 53.7 85.0j– – EXEM [9] G N 77.2 59.8 – – j69.6 – SCoRe [39] V N 82.8 59.5 – – j– – LESD [11] V/G N 82.8 56.2 58.8 88.3j”
Zero-Shot Sketch-Image Hashing (2018, background): “Zero-shot learning [19, 31, 75, 76, 56, 46, 7, 2, 64, 3, 34, 11, 8, 25, 74, 14, 35, 65, 27, 68, 40] is also related to our work, though it does not originally focus on cross-modal retrieval.”
Learning Embedding Adaptation for Few-Shot Learning (2018, background): “Another line of approach has focused on learning generalizable instance embeddings [1, 5, 6, 18, 28, 42, 44] and uses those embeddings on simple classifiers such as nearest neighbor classification rules.”
Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs (2018, methodology): “We compare our results to the recent state-of-theart methods SYNC [4] and EXEM [5].”
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “Depending on the selection of joint embedding space and type of projection function utilised between the visual to semantic space, existing models can be divided into three groups: (i) projected from visual feature space to semantic space [13, 20], (ii) projected from semantic space to the visual fe”