Latent Embeddings for Zero-Shot Classification
By Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh N. Nguyen, Matthias Hein, Bernt Schiele. 2016
750 citations recorded by the source corpus.
We present a novel latent embedding model for learning a compatibility function between image and class embeddings, in the context of zero-shot classification. The proposed method augments the state-of-the-art bilinear compatibility model by incorporating latent variables. Instead of learning a single bilinear map, it learns a collection of maps with the selection, of which map to use, being a latent variable for the current image-class pair. We train the model with a ranking based objective function which penalizes incorrect rankings of the true class for a given image. We empirically demonstrate that our model improves the state-of-the-art for various class embeddings consistently on three challenging publicly available datasets for the zero-shot setting. Moreover, our method leads to visually highly interpretable results with clear clusters of different fine-grained object properties that correspond to different latent variable maps.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Trevor Darrell
- Bernt Schiele
- Jianfeng Gao
- Mohit Bansal
- Zeynep Akata
- Dacheng Tao
- Yoshua Bengio
- Luc Van Gool
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Bernt Schiele
- Ming-Hsuan Yang
- Yi Yang
- Jitendra Malik
In Radiology, Nuclear Medicine and Imaging
- Zeynep Akata
- Shafin Rahman
- Bernt Schiele
- Yongqin Xian
- Fei Sha
- Trevor Darrell
- Salman Hameed Khan
- Matthieu Cord
In Representation & Self-Supervised Learning
- Trevor Darrell
- Mingsheng Long
- Jianmin Wang
- Boqing Gong
- Kate Saenko
- Fei Sha
- Zeynep Akata
- Bernt Schiele
In Meta & Continual Learning
- Shafin Rahman
- Salman Hameed Khan
- Bernt Schiele
- Venkatesh Saligrama
- Hanxiao Wang
- Pengkai Zhu
- Zeynep Akata
- Matthieu Cord
Citations in the corpus
- Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “In this sense, our work is also related to conventional zero-shot classification [9, 28, 45, 62, 65, 66].”
- Learning to Compare: Relation Network for Few-Shot Learning (2017, neutral): “6 LATEM [41] 55.”
- Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “The analysis of the latent space organization is becoming crucial towards understanding and improvement of classification models [68, 49].”
- Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning (2021, background): “Being inspired by the literature on zero-shot learning [7, 6, 5, 20, 37, 24, 25], given an image as input, we estimate the semantic word vectors for the input instead of directly predicting its class label.”
- Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, methodology): “Latent Embeddings (LATEM) [11] and Cross Modal Transfer (CMT) [12] encode an additional non-linearity component to linear compatibility learning framework.”
- Weakly-Supervised Learning of Visual Relations (2017, background): “Zero-shot learning has been mostly explored for object classification [13, 29, 46, 48] and recently for the task of describing images with novel objects [19, 47].”
- Zero-Shot Object Detection (2018, background): “In [52], the authors augment the bilinear compatibility model by adding latent variables.”
- Feature Generating Networks for Zero-Shot Learning (2017, methodology): “[50, 31, 8] learn unseen classes as a mixture of seen class proportions, and [2, 3, 14, 42, 45, 40, 16, 33, 1, 6, 17, 23] learn a compatibility between images and classes.”
- Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders (2018, methodology): “On the other hand, the classic ZSL methods DeViSE [6], SJE [2], ALE [1], EZSL [24] and LATEM [34] use a linear compatibility function or other similarity metrics to compare embedded visual and semantic features; CMT [27] and LATEM [34] utilize multiple neural networks to learn a non-linear embedding”
- Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects (2019, methodology): “Alternatively, some methods attempt to find a latent intermediate space to calculate the matching score [40].”
- Zero-Shot Sketch-Image Hashing (2018, background): “Zero-shot learning [19, 32, 74, 75, 55, 46, 7, 2, 63, 3, 35, 11, 8, 26, 73, 14, 36, 64, 28, 67, 40] is also related to our work, though it does not originally focus on cross-modal retrieval.”
- Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning (2016, methodology): “Multiple types of semantic information have been exploited in the literature: visual attributes [11, 17], word vector representations of class names [12, 39, 27], textual descriptions [10, 19, 32], hierarchical ontology of classes (such as WordNet [26]) [2, 21, 45], and human gazes [15].”
- PointCLIP: Point Cloud Understanding by CLIP (2021, background): “Although zeroshot learning has drawn much attention on 2D classification [27, 46, 59], only a few works explore how to conduct it in 3D domain.”
- Zero Shot Detection (2018, background): “Zero Shot Learning (ZSL) seeks to recognize novel visual categories that are unannotated in training data [4], [12], [13], [30].”
- Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Many other ZSL approaches learn nonlinear multi-modal embedding [45, 2, 53, 6, 64], where most of the methods focus to learn a non-linear mapping from the image space to the semantic space.”
- Zero-Shot Learning via Joint Latent Similarity Embedding (2015, background): “With this in mind, we predict a class label y(t) for target domain instance x(t) ∈ X (t).”
- Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks (2017, methodology): “To scale up ZSL, embedding based methods are prevailing [14, 2, 3, 49, 61, 55, 25, 31].”