Zero-Shot Learning via Semantic Similarity Embedding

By Ziming Zhang, Venkatesh Saligrama. 2015

649 citations recorded by the source corpus.

In this paper we consider a version of the zero-shot learning problem where seen class source and target domain data are provided. The goal during test-time is to accurately predict the class label of an unseen target domain instance based on revealed source domain side information (e.g. attributes) for unseen classes. Our method is based on viewing each source or target data as a mixture of seen class proportions and we postulate that the mixture patterns have to be similar if the two instances belong to the same unseen class. This perspective leads us to learning source/target embedding functions that map an arbitrary source/target domain data into a same semantic space where similarity can be readily measured. We develop a max-margin framework to learn these similarity functions and jointly optimize parameters by means of cross validation. Our test results are compelling, leading to significant improvement in terms of accuracy on most benchmark datasets for zero-shot recognition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Compare: Relation Network for Few-Shot Learning (2017, methodology): “When applied to ZSL our architecture is related to methods that learn to align images and category embeddings and perform recognition by predicting if an image and category embedding pair match [11, 3, 43, 46].”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, methodology): “SSE [13] leverages similar class relationships both in image and semantic embedding space.”
Zero-Shot Learning — The Good, the Bad and the Ugly (2017, methodology): “In addition to [22, 42, 39, 4, 32, 7, 34], we re-implement [26, 11, 3] based on the original publications.”
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild (2016, methodology): “Existing research on ZSL has been almost entirely focusing on this setting [4,10,11,12,13,14,15,8,16,17,18,19,20,21,22,23,24,25,26,27,28].”
Feature Generating Networks for Zero-Shot Learning (2017, background): “[50, 31, 8] learn unseen classes as a mixture of seen class proportions, and [2, 3, 14, 42, 45, 40, 16, 33, 1, 6, 17, 23] learn a compatibility between images and classes.”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “[42] use a Semantic Similarity Embedding (SSE) in order to perform action recognition.”
Identifying Ambiguous Similarity Conditions via Semantic Matching (2022, methodology): “CSL has been applied in applications like image-text classification [20,24], fashion retrieval [8,15,32], zero-shot learning [48] and video grounding [28].”
Zero-Shot Sketch-Image Hashing (2018, methodology): “However, existing works basically focus on single-modal zero-shot recognition [56, 75, 76, 31], and are not suitable for efficient image retrieval.”
Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts (2018, background): “Since its inception, zero-shot learning research has been dominated by the object classification problem [2,5,10,18,21,22,29,33,39,47,51,52,53].”
A Survey of Zero-Shot Learning (2019, background): “For example, in SSE [173], in the training phase, each seen class is represented as a combination of all seen classes.”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “Mapping both image and semantic features into another common intermediate space is another direction that ZSL approaches adapt [66, 16, 67, 1, 28].”
Learning a Deep Embedding Model for Zero-Shot Learning (2016, methodology): “Inspired by humans’ ZSL ability, recently there is a surge of interest in machine ZSL [2, 47, 22, 1, 37, 43, 10, 31, 11, 14, 24, 46, 34, 4, 13, 3, 5, 48, 49].”
Learning Transferable Knowledge Through Embedding Spaces (2019, methodology): “Numerous ZSL algorithms have been developed to learn a mapping between the visual features and semantic attributes using a shared space [52; 53; 54; 55; 56; 19].”
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “Depending on the selection of joint embedding space and type of projection function utilised between the visual to semantic space, existing models can be divided into three groups: (i) projected from visual feature space to semantic space [13, 20], (ii) projected from semantic space to the visual fe”
Zero-shot Image Recognition Using Relational Matching, Adaptation and Calibration (2019, background): “AwA2 aPY CUB SUN Method tr u s H tr u s H tr u s H tr u s H DAP [3] 46.1 0.0 84.7 0.0 33.8 4.8 78.3 9.0 40.0 1.7 67.9 3.3 39.9 4.2 25.1 7.2 IAP [3] 35.9 0.9 87.6 1.8 36.6 5.7 65.6 10.4 24.0 0.2 72.8 0.4 19.4 1.0 37.8 1.8 CONSE [23] 44.5 0.5 90.6 1.0 26.9 0.0 91.2 0.0 34.3 1.6 72.2 3.1 38.8 6.8 39.9 ”
Zero-Shot Learning via Joint Latent Similarity Embedding (2015, methodology): “This literature includes [3, 16, 44], where [16, 44] embed source and target domain data nonlinearly into known semantic spaces (i.”
Region Graph Embedding Network for Zero-Shot Learning (2020, methodology): “Most of the aforementioned methods adopt the following scheme for ZSL: 1) extracting global features from pre-trained [22, 62, 29, 72, 73] or end-to-end trainable nets [33, 42, 26] (Fig.1(a)) and 2) constructing embedding or generative models by associating these features with their semantic…”
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (2020, positive): “And the classic zero-shot image recognition techniques [1, 3, 6, 18, 29, 35, 47, 52, 54, 61, 65] are directly applied to learn from seen classes the visual-semantic mappings, which are then transferred to unseen ones.”
Synthesized Classifiers for Zero-Shot Learning (2016, methodology): “Finally, classifiers for the unseen classes can directly be constructed in the input feature space [1, 8, 23, 27, 44, 47].”
Leveraging the Invariant Side of Generative Zero-Shot Learning (2019, methodology): “In addition, there are also some works to learn an intermediate space shared by the visual features and semantic features [4, 38, 39].”