Zero-Shot Learning Through Cross-Modal Transfer

By Richard Socher, Milind Ganjoo, Christopher D. Manning, Andrew Y. Ng. 2013

1,533 citations recorded by the source corpus.

This work introduces a model that can recognize objects in images even if no training data is available for the object class. The only necessary knowledge about unseen visual categories comes from unsupervised text corpora. Unlike previous zero-shot learning models, which can only differentiate between unseen classes, our model can operate on a mixture of seen and unseen classes, simultaneously obtaining state of the art performance on classes with thousands of training images and reasonable performance on unseen classes. This is achieved by seeing the distributions of words in texts as a semantic space for understanding what objects look like. Our deep learning model does not require any manually defined semantic or visual features for either words or images. Images are mapped to be close to semantic word vectors corresponding to their classes, and the resulting image embeddings can be used to distinguish whether an image is of a seen or unseen class. We then use novelty detection methods to differentiate unseen classes from seen classes. We demonstrate two novelty detection strategies; the first gives high accuracy on unseen classes, while the second is conservative in its prediction of novelty and keeps the seen classes' accuracy high.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “To do so, we used the sentences from the Stanford Sentiment Treebank dataset (Socher et al., 2013) and rendered them into images, with black texts on a white background, in a 448×448 resolution.”
Shaping Visual Representations with Language for Few-Shot Classification (2019, neutral): “Language has been shown to assist visual classiﬁcation in various settings, including traditional visual classiﬁcation with no transfer [16] and with language available at test time in the form of class labels or descriptions for zero-[10, 11, 27] or few-shot [24, 33] learning.”
Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “Our work is also related to prior studies on multi-modal embedding [44, 56] that explore approaches such as kernel canonical component analysis [16, 18], restricted Boltzmann machines [45], topic models [22], and log-bilinear models [25].”
Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions (2015, methodology): “The Euclidean distance loss function was previously used in [26] with a fixed pre-learnt word embedding.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Related work has also been presented at NIPS (Socher et al., 2013b; Frome et al., 2013).”
Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search (2021, methodology): “trained a model to map images in feature vectors close to semantic word vectors corresponding to their class and the model showed some zero shot capabilities(Socher et al., 2013).”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “…between images and texts, which is related to many previous works, such as (Hironobu et al., 1999; Weston et al., 2010; Socher and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a;…”
Learning to Prompt for Vision-Language Models (2021, methodology): “embedding space has been studied since nearly a decade ago (Socher et al., 2013; Frome et al., 2013; Elhoseiny et al., 2013), but with drastically different technologies.”
Learning feed-forward one-shot learners (2016, background): “Whereas it is usually framed as a modality transfer problem and solved through transfer learning [21], Noh et al.”
Rewriting a Deep Generative Model (2020, background): “Examples include domain adaptation [66], zero-shot or few-shot learning [71,51], model pre-training and feature learning [17,82,78], and meta-learning [8,4,23].”
Look, Listen and Learn (2017, neutral): “ormation. In our case the two views are visual and audio (and each can determine semantic information independently). A similar scenario arises when the two views are visual and language (text) as in [9,18,31] where a common embedding is learnt. However, usually one (or both) of the networks (for im”
Learning Deep Representations of Fine-Grained Visual Descriptions (2016, background): “for zero-shot learning [36, 38, 24, 33, 12, 41, 2] and automatically generating image captions for general web images [23, 35, 45, 20, 8].”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, methodology): “In zero-shot learning (Larochelle et al., 2008; Socher et al., 2013), we are typically given training images with labels and some additional intermediate semantic representation T , such as textual descriptions.”
Towards Open Set Deep Networks (2015, background): “Or detection can used as a flag to bring in other modalities [24, 7].”
Deep Learning: Methods and Applications (2014, background): “[342] map images to single-word vectors so that the constructed multi-modal system can classify images without seeing any examples of the class, i.”
Transductive Multi-View Zero-Shot Learning (2015, methodology): “Without any labelled data for the target classes, exter-nal knowledge is needed to represent what each target class looks like, in the form of class prototypes.”