Evaluation of output embeddings for fine-grained image classification

By Zeynep Akata, Scott E. Reed, Daniel Walter, Honglak Lee, Bernt Schiele. 2014

1,086 citations recorded by the source corpus.

Image classification has advanced significantly in recent years with the availability of large-scale image sets. However, fine-grained classification remains a major challenge due to the annotation cost of large numbers of fine-grained categories. This project shows that compelling classification performance can be achieved on such categories even without labeled training data. Given image and class embeddings, we learn a compatibility function such that matching embeddings are assigned a higher score than mismatching ones; zero-shot classification of an image proceeds by finding the label yielding the highest joint compatibility score. We use state-of-the-art image features and focus on different supervised attributes and unsupervised output embeddings either derived from hierarchies or learned from unlabeled text corpora. We establish a substantially improved state-of-the-art on the Animals with Attributes and Caltech-UCSD Birds datasets. Most encouragingly, we demonstrate that purely unsupervised output embeddings (learned from Wikipedia and improved with finegrained text) achieve compelling results, even outperforming the previous supervised state-of-the-art. By combining different output embeddings, we further improve results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “This method learns a compatibility function of image and text, which can be seen as an extension of the multimodal structured jointed embedding [35].”
Generative Adversarial Text to Image Synthesis (2016, methodology): “, 2014), in particular to enable zero-shot visual recognition (Fu et al., 2014; Akata et al., 2015), and recently for conditional image generation (Yan et al.”
Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “…and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et al., 2020; Chen et al.,…”
Prototypical Networks for Few-shot Learning (2017, methodology): “We compare our method to other embedding approaches, such as ALE [1], SJE [2], and DS-SJE/DA-SJE [23].”
Learning What and Where to Draw (2016, methodology): “To train the text encoder we minimize a surrogate loss related to Equation 1 (see Akata et al. [2015] for details).”
Learning Deep Representations of Fine-Grained Visual Descriptions (2016, methodology): “All neural text model architectures are the same as we used for CUB, and we used the same hyperparameters from crossvalidation on CUB.”
Learning to Compare: Relation Network for Few-Shot Learning (2017, methodology): “When applied to ZSL our architecture is related to methods that learn to align images and category embeddings and perform recognition by predicting if an image and category embedding pair match [11, 3, 43, 46].”
Zero-Shot Learning — The Good, the Bad and the Ugly (2017, methodology): “For sanity-check, we re-evaluate methods [21, 42, 39, 4, 31, 7] 2 using provided features and code.”
Neural Data Augmentation via Example Extrapolation (2021, background): “…problem for tasks with constantly expanding output spaces, such as relation extraction (Han et al., 2018) and entity linking (Logeswaran et al., 2019), or particularly long-tail output spaces, such as fine-grained ∗ Equal contribution from all authors. image classification (Akata et al., 2015).”
Zero-Shot Object Detection (2018, neutral): “More recent approaches have used multimodal embeddings to learn a compatibility function between an image vector and class label embeddings [1,2].”
Latent Embeddings for Zero-Shot Classification (2016, methodology): “Due to the bilinearity of F in x and y, previous work [1, 2, 38] refer to this model as a bilinear model, however one can also view it as a linear one since F is linear in the parameter W.”
Label-Embedding for Image Classification (2015, positive): “Such an embedding , which we refer to as Word2Vec Label Embedding (WLE) , was recently used for zero-shot recognition [20] on ﬁne-grained datasets [2].”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, methodology): “We compare the learned attributes by our model with Word2Vec feature vectors (Mikolov et al., 2013) extracted from Wikipedia articles (Akata et al., 2015).”
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild (2016, methodology): “Existing research on ZSL has been almost entirely focusing on this setting [4,10,11,12,13,14,15,8,16,17,18,19,20,21,22,23,24,25,26,27,28].”
Feature Generating Networks for Zero-Shot Learning (2017, methodology): “[50, 31, 8] learn unseen classes as a mixture of seen class proportions, and [2, 3, 14, 42, 45, 40, 16, 33, 1, 6, 17, 23] learn a compatibility between images and classes.”
Continual Representation Learning for Biometric Identification (2020, background): “Second, biometric identification focuses on a more challenging setting which is similar to fine-grained classification [2,64,70,73].”
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders (2018, methodology): “On the other hand, the classic ZSL methods DeViSE [6], SJE [2], ALE [1], EZSL [24] and LATEM [34] use a linear compatibility function or other similarity metrics to compare embedded visual and semantic features; CMT [27] and LATEM [34] utilize multiple neural networks to learn a non-linear embedding”
Generating Visual Representations for Zero-Shot Classification (2017, background): “, [2, 6, 5, 32] are focused on improving the embedding or the scoring function.”
Opening up Open World Tracking (2021, methodology): “To glue known and unknown visual concepts, ZSL methods utilize meta-information in form of attributes [38, 2, 71, 51] or leverage word embedding models [95, 1].”