Label-Embedding for Image Classification

By Zeynep Akata, Florent Perronnin, Zaïd Harchaoui, Cordelia Schmid. 2015

856 citations recorded by the source corpus.

Attributes act as intermediate representations that enable parameter sharing between classes, a must when training data is scarce. We propose to view attribute-based image classification as a label-embedding problem: each class is embedded in the space of attribute vectors. We introduce a function that measures the compatibility between an image and a label embedding. The parameters of this function are learned on a training set of labeled samples to ensure that, given an image, the correct classes rank higher than the incorrect ones. Results on the Animals With Attributes and Caltech-UCSD-Birds datasets show that the proposed framework outperforms the standard Direct Attribute Prediction baseline in a zero-shot learning scenario. Label embedding enjoys a built-in ability to leverage alternative sources of information instead of or in addition to attributes, such as, e.g., class hierarchies or textual descriptions. Moreover, label embedding encompasses the whole range of learning settings from zero-shot learning to regular learning with a large number of labeled examples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, methodology): “…that can generally be applied to any classifier, or have been used on traditional classifiers such as support vector machines or decision trees (Akata, 2013; Akata, et al., 2015; Bilgic and Mooney, 2005; Biran and McKeown, 2014, 2017; Craven and Shavlik, 1996; Lakkaraju, Kamar, Caruana, and…”
Joint Embedding of Words and Labels for Text Classification (2018, background): “In computer vision, there has been a vast amount of research on leveraging label embeddings for image clas-siﬁcation (Akata et al., 2016), multimodal learning between images and text (Frome et al., 2013; Kiros et al., 2014), and text recognition in images (Rodriguez-Serrano et al., 2013).”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, methodology): “Although Direct Attribute Prediction (DAP) [1] and Indirect At-tribute Prediction (IAP) [1] have been shown to perform poorly compared to compatibility learning frameworks [30], we include them to our evaluation for being historically the most widely used methods in the literature.”
Feature Generating Networks for Zero-Shot Learning (2017, background): “Figure 6 shows that softmax as a classiﬁer obtains the state-of-the-art of ZSL and GZSL on ImageNet, signiﬁcantly improving over ALE [2].”
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders (2018, methodology): “On the other hand, the classic ZSL methods DeViSE [5], SJE [2], ALE [1], EZSL [22] and LATEM [33] use a linear compatibility function or other similarity metrics to compare embedded visual and semantic features;”
Zero-Shot Detection via Vision and Language Knowledge Distillation (2021, background): “Prior to the breakthrough of deep learning, earlier work study the use of attributes to encode categories as vectors of binary attributes [22, 10, 38] and learn label embeddings [2].”
Zero-Shot Object Detection: Joint Recognition and Localization of Novel Concepts (2020, methodology): “b) The second kind of methods learn a linear [2,3,47] or non-linear [55,52] compatibility function to relate the seen image feature and corresponding seman-”
Baby steps towards few-shot learning with multiple semantics (2019, background): “The semantic attention for each branch is a two-layer MLP with a 300-sized hidden layer and a scalar output layer followed by a sigmoid (to normalize the coefficients into a [0, 1] range).”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, background): “For example, [1,47] proposed to learn a joint embedding space between the images and the name/description of the category for image classification, and [28] explored taking the advantages of mid-level semantic representation.”
Transductive Zero-Shot Learning for 3D Point Cloud Classification (2019, methodology): “Zero-Shot Learning: For the ZSL task, there has been significant progress, including on image recognition [35, 65, 2, 4, 29, 21, 57], multi-label ZSL [22, 34], and zero-shot detection [36].”
Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects (2019, background): “Zero-Shot Learning (ZSL) on 2D images: In the image recognition literature, zero-shot learning has made reasonable progress over the past few years [3, 4, 17, 24, 30, 41, 44].”
Leveraging the Invariant Side of Generative Zero-Shot Learning (2019, methodology): “Speciﬁcally, we compare our approach with: DAP [15], CONSE [26], SSE [38], DeViSE [7], SJE [2], ESZSL [29], ALE [1], SYNC [4], SAE [13], DEM [37], GAZSL [40] and f-CLSWGAN [34].”