Zero-Shot Learning by Convex Combination of Semantic Embeddings

By Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S. Corrado, Joseph Dean. 2013

981 citations recorded by the source corpus.

Abstract: Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the embedding space is trained jointly with the image transformation. In other cases the semantic embedding space is established by an independent natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional \nway{} classification framing of image understanding, particularly in terms of the promise for zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. In this paper, we propose a simple method for constructing an image embedding system from any existing \nway{} image classifier and a semantic word embedding model, which contains the $\n$ class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional training. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Matching Networks for One Shot Learning (2016, background): “In this case, a label set L sampled from a task T , L ∼ T , will typically have 5 to 25 examples.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, neutral): “GQA As shown in Table 9, compared with existing VLP works (LXMERT [37] and 12-in-1 [24]), our OscarB results on GQA gain 0.”
Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions (2015, methodology): “Given convolutional feature a and fully connected feature x after propagating the raw image through the CNN, we can write down the joint classification model as: ŷc = w T c gv(x) + o ( K′∑ i=1 w′c,i ∗̌ g′v(a)i ) .”
Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “…works, such as (Hironobu et al., 1999; Weston et al., 2010; Socher and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et…”
Order-Embeddings of Images and Language (2015, methodology): “Previous approaches, including Frome et al. (2013) and Norouzi et al. (2014) have embedded words and images into a shared semantic space with symmetric similarity—which our experiments suggest to be a poor ﬁt with the partial order structure of WordNet.”
Learning Deep Representations of Fine-Grained Visual Descriptions (2016, methodology): “for zero-shot learning [36, 38, 24, 33, 12, 41, 2] and automatically generating image captions for general web images [23, 35, 45, 20, 8].”
Learning to Compare: Relation Network for Few-Shot Learning (2017, neutral): “3 CONSE [28] 45.”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, background): “CONSE [15] learns the probability of a training image belonging to a training class: where y denotes the most likely training label ( t =1) for image x .”
Taskonomy: Disentangling Task Transfer Learning (2018, background): “compositional modeling [35, 10, 13, 23, 55, 92, 90], homomorphic cryptography [42], lifelong learning [93, 15, 85, 84], functional maps [71], certain aspects of Bayesian inference and Dirichlet processes [54, 91, 90, 89, 37, 39], few-shot learning [81, 25, 24, 70, 86], transfer learning [75, 84, 29,”
Learning from Task Descriptions (2020, background): “Zero-shot learning (Chang et al., 2008; Socher et al., 2013; Norouzi et al., 2013) asks systems to generalize to unseen classes at test time.”
Zero-Shot Learning — The Good, the Bad and the Ugly (2017, methodology): “In addition to [22, 42, 39, 4, 32, 7, 34], we re-implement [26, 11, 3] based on the original publications.”
Evaluation of output embeddings for fine-grained image classification (2014, methodology): “ConSe [37] uses the probabilities of a softmaxoutput layer to weigh the semantic vectors of all the classes.”
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation (2021, background): “Frome et al. (2013) and Norouzi et al. (2014) are pioneering works that learn a visual-semantic embedding space using deep learning.”
Exploring the structure of a real-time, arbitrary neural artistic stylization network (2017, background): “One might leverage metadata of paintings in order to refine the embedding representation through a secondary embedding loss [7, 20].”
Latent Embeddings for Zero-Shot Classification (2016, methodology): “Such label embedding methods [1, 2, 12, 13, 25, 26, 32, 35] aim to find a mapping between two embedding spaces, one each for the two streams of information e.”
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild (2016, methodology): “[4], this setting has almost always been adopted for evaluating ZSL methods [10,11,12,13,14,15,8,16,17,18,19,20,21,22,23,24,25,26,27,28].”