Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions

By Jimmy Ba, Kevin Swersky, Sanja Fidler, Ruslan Salakhutdinov. 2015

452 citations recorded by the source corpus.

One of the main challenges in Zero-Shot Learning of visual categories is gathering semantic attributes to accompany images. Recent work has shown that learning from textual descriptions, such as Wikipedia articles, avoids the problem of having to explicitly define these attributes. We present a new model that can classify unseen categories from their textual description. Specifically, we use text features to predict the output weights of both the convolutional and the fully connected layers in a deep convolutional neural network (CNN). We take advantage of the architecture of CNNs and learn features at different layers, rather than just learning an embedding space for both modalities, as is common with existing approaches. The proposed model also allows us to automatically generate a list of pseudo-attributes for each visual category consisting of words from Wikipedia articles. We train our models end-to-end using the Caltech-UCSD bird and flower datasets and evaluate both ROC and Precision-Recall curves. Our empirical results show that the proposed model significantly outperforms previous methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Lei Ba et al. (2015) first introduced a zero-shot image classifier of this form while the idea of generating a classifier from natural language dates back to at least Elhoseiny et al. (2013).”
ExpBERT: Representation Engineering with Natural Language Explanations (2020, methodology): “parameters of a classifier (Ba et al., 2015) and as part of the parameter space of a classifier (Andreas et al.”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, methodology): “Other works map directly from text to parameters in models (Ba et al., 2015; Andreas et al., 2018), in effect learning a prior p(θ|text) (though Andreas et al.”
Learning to Prompt for Vision-Language Models (2021, methodology): “For text features extraction, early work has mainly utilized pre-trained word vectors (Socher et al., 2013; Frome et al., 2013) or the hand-crafted TF-IDF features (Elhoseiny et al., 2013; Lei Ba et al., 2015).”
Prototypical Networks for Few-shot Learning (2017, methodology): “With respect to zero-shot learning, the use of embedded meta-data in Prototypical Networks resembles the method of [3] in that both predict the weights of a linear classifier.”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “Another strand of research on multi-modal embeddings is based on deep learning [3, 24, 25, 31, 35, 44], utilizing such techniques as deep Boltzmann machines [44], autoencoders [35], LSTMs [8], and recurrent neural networks [31, 45].”
Look, Listen and Learn (2017, neutral): “ormation. In our case the two views are visual and audio (and each can determine semantic information independently). A similar scenario arises when the two views are visual and language (text) as in [9,18,31] where a common embedding is learnt. However, usually one (or both) of the networks (for im”
Learning Deep Representations of Fine-Grained Visual Descriptions (2016, methodology): “For instance on the Caltech-UCSD birds database (CUB) [46], previous zero-shot learning approaches [13, 2, 3] have used human-encoded attributes [24], or simplified language models such as bag-of-words [16], WordNet-hierarchyderived features [29], and neural word embeddings such as Word2Vec [28] and”
Where to Look: Focus Regions for Visual Question Answering (2015, background): “Yu et al. [23], which proposes a Visual Madlibs dataset for ﬁll-in-the-blank and question answering, focuses their approach on learning latent embeddings and ﬁnds normalized CCA on averaged word2vec representations [10, 16] to outperform recurrent networks for embedding.”
Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction (2015, methodology): “The prediction of the weight parameters in deep neural networks has been explored in [2] in the context of zeroshot learning.”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, background): “[31], [34], [50], [51], [52], [53], [54] which exploit other auxiliary information that reduces this annotation effort.”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, methodology): “[50] trained a twobranch network using classification loss to match visual and text data for zero-shot learning.”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “This approach has been shown to be effective for achieving zero-shot and one-shot generalization in image classiﬁcation problems (Lei Ba et al., 2015; Bertinetto et al., 2016).”
Conditional Prompt Learning for Vision-Language Models (2022, methodology): “Images are often encoded using hand-crafted descriptors [10,45] or neural networks [12,29], while texts are encoded using, for instance, pre-trained word vectors [12, 45] or the frequency-based TF-IDF features [10,29].”
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild (2016, background): “[33] and [34] focus on training a zero-shot binary classifier for each unseen class (against seen ones) — it is not clear how to distinguish multiple unseen classes from the seen ones.”
ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases (2017, background): “Deep neural network representations further make the joint language and vision learning tasks more feasible to solve, in image captioning [47, 23, 32, 46, 22], visual question answering [2, 45, 49, 53] and knowledge-guided transfer learning [4, 33], and so on.”
Don’t Even Look Once: Synthesizing Features for Zero-Shot Detection (2019, background): “Zero Shot Learning (ZSL) seeks to recognize novel visual categories that are unannotated in training data [21, 40, 22, 45].”