Write a Classifier: Predicting Visual Classifiers from Unstructured Text

By Mohamed Elhoseiny, Ahmed M. Elgammal, Babak Saleh. 2015

42 citations recorded by the source corpus.

People typically learn through exposure to visual concepts associated with linguistic descriptions. For instance, teaching visual object categories to children is often accompanied by descriptions in text or speech. In a machine learning context, these observations motivates us to ask whether this learning process could be computationally modeled to learn visual classifiers. More specifically, the main question of this work is how to utilize purely textual description of visual classes with no training images, to learn explicit visual classifiers for them. We propose and investigate two baseline formulations, based on regression and domain transfer, that predict a linear classifier. Then, we propose a new constrained optimization formulation that combines a regression function and a knowledge transfer function with additional constraints to predict the parameters of a linear classifier. We also propose a generic kernelized models where a kernel classifier is predicted in the form defined by the representer theorem. The kernelized models allow defining and utilizing any two Reproducing Kernel Hilbert Space (RKHS) kernel functions in the visual space and text space, respectively. We finally propose a kernel function between unstructured text descriptions that builds on distributional semantics, which shows an advantage in our setting and could be useful for other applications. We applied all the studied models to predict visual classifiers on two fine-grained and challenging categorization datasets (CU Birds and Flower Datasets), and the results indicate successful predictions of our final model over several baselines that we designed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Efficient Lifelong Learning with A-GEM (2018, background): “, 2017), and it is also a key ingredient in all the zero/few-shot learning algorithms (Lampert et al., 2014; Xian et al., 2018; Elhoseiny et al., 2017; Wah et al., 2011; Lampert et al., 2009).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Progress in zero-shot learning, where the aim is to identify or categorise images for which little or no training data is available, is likely to contribute to the resolution of data sparseness problems (e.g. Antol et al., 2014; Elhoseiny et al., 2017).”
Choose Your Neuron: Incorporating Domain Knowledge through Neuron-Importance (2018, methodology): “vectors of activations from some late layer in a network pretrained on some largescale task) to learn joint embeddings with class descriptions [32,1,3,5,23,8,9,7].”
Efficient Lifelong Learning with A-GEM (2018, background): “, 2017), and it is also a key ingredient in all the zero/few-shot learning algorithms (Lampert et al., 2014; Xian et al., 2018; Elhoseiny et al., 2017; Wah et al., 2011; Lampert et al., 2009).”
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts (2017, unknown): “0 – – WAC-Kernel [8] 33.”
Small Sample Learning in Big Data Era (2018, methodology): “Inspired by (Elhoseiny et al., 2013, 2017a) to learn pseudo-concepts to associate novel classes using Wikipedia articles, (Ba et al., 2015) used text features to predict the output weights of both the convolutional and the fully connected layers in a deep CNN as visual features.”