Semi-supervised Vocabulary-Informed Learning

By Yanwei Fu, Leonid Sigal. 2016

135 citations recorded by the source corpus.

Despite significant progress in object categorization, in recent years, a number of important challenges remain, mainly, ability to learn from limited labeled data and ability to recognize object classes within large, potentially open, set of labels. Zero-shot learning is one way of addressing these challenges, but it has only been shown to work with limited sized class vocabularies and typically requires separation between supervised and unsupervised classes, allowing former to inform the latter but not vice versa. We propose the notion of semi-supervised vocabulary-informed learning to alleviate the above mentioned challenges and address problems of supervised, zero-shot and open set recognition using a unified framework. Specifically, we propose a maximum margin framework for semantic manifold-based recognition that incorporates distance constraints from (both supervised and unsupervised) vocabulary atoms, ensuring that labeled samples are projected closest to their correct prototypes, in the embedding space, than to others. We show that resulting model shows improvements in supervised, zero-shot, and large open set recognition, with up to 310K class vocabulary on AwA and ImageNet datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Natural Language Processing

Citations in the corpus

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, neutral): “They are usually in single domain, with a manual split of categories to product disjoint training and test categories, e.g., Animal with Attributes (AwA) [31], Caltech-UCSD Birds-200 (CUB) [64], SUN [47], aPY [14], and ZS-ImageNet [52, 17].”
Learning to Compare: Relation Network for Few-Shot Learning (2017, background): “cent methods [48, 12, 13] are tranductive in that they use all test data at once for model training, which gives them a big advantage at the cost of making a very strong assumption that may not be met in practical applications, so we do not compare with them here.”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, methodology): “[46], [47], [48] use semi-supervised learning based on max-margin framework.”
Zero-Shot Learning — The Good, the Bad and the Ugly (2017, neutral): “compatibility learning frameworks [3, 4, 11, 15, 27, 32, 34, 39, 32, 12, 29, 1, 6, 24, 13, 21].”
Recent Advances in Open Set Recognition: A Survey (2018, background): “Besides, some related works can be found in [132], [153], [161], [162].”
Feature Generating Networks for Zero-Shot Learning (2017, background): “[50, 31, 8] learn unseen classes as a mixture of seen class proportions, and [2, 3, 14, 42, 45, 40, 16, 33, 1, 6, 17, 23] learn a compatibility between images and classes.”
Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy (2019, methodology): “AMP [6] 13.1 SS-Voc [5] 16.8 DeViSE [4] 12.8 ConSE [18] 15.5 VZSL [29] 23.1 CVAE [17] 24.7 DEM [32] 25.7 SAE [11] 27.2 Ours 27.9 mapping-learning-based ZSL model [11] demonstrate that the proposed model is more suitable for alleviating the domain gap issue in large-scale ZSL.”
Unseen Class Discovery in Open-world Classification (2018, background): “Fu & Sigal (2016) solves a similar problem of detecting unseen class examples and classifying them by leveraging image description vocabulary and assuming unseen class examples’ labels are available in the vocabulary.”
Gradient Matching Generative Networks for Zero-Shot Learning (2019, background): “For example, [6] models the joint probability of attributes, [8] models the conditional distribution of features given attributes, [33] uses semantic knowledge bases for attribute classification, [34, 35, 36] build convex combinations of seen class classifiers, [15, 17, 18, 19, 20, 21, 22] learn a c”
Delta-encoder: an effective sample synthesis method for few-shot object recognition (2018, methodology): “fine tuning from pre-trained models [25, 45, 46]; (3) applying domain transfer by borrowing examples from relevant categories [27] or using semantic vocabularies [2, 12]; (4) rendering synthetic examples [32, 7, 40]; (5) augmenting the training examples [23]; (6) example synthesis using Generative A”
A Generative Model for Zero Shot Learning Using Conditional Variational Autoencoders (2017, methodology): “However, Imagenet has no such annotated features and we use 1000 dimensional word2vec features [27] trained using a skip-gram model on the Wikipedia corpus, similar to [13].”
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts (2017, background): “The core of such approaches is to (a) learn a mapping function from the visual feature space to the semantic space [41, 11, 12], or conversely [51, 39], (b) or jointly learn the embedding function between the visual and semantic space through a latent space [46, 22, 3, 36, 1].”
Learning a Deep Embedding Model for Zero-Shot Learning (2016, methodology): “This is why the word vector semantic space is still attractive: it is ‘free’ and is the only choice for large scale recognition with many unseen classes [13].”
Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs (2018, background): “The core idea is to represent each category with learned vector representations that can be mapped to visual classifiers [48, 44, 13, 41, 25, 15, 14, 18, 23, 4, 5, 55, 54].”
Zero-Shot Learning via Class-Conditioned Deep Generative Models (2017, methodology): “For the large-scale dataset (ImageNet), we follow (Fu et al. 2016), for which 1000 classes from ILSVRC2012 (Rus-sakovsky et al. 2015) are used as seen classes, while 360 non-overlapped classes of ILSVRC2010 (Deng et al. 2009 (Simonyan et al. 2014) as our raw input representation, which is a…”
Semantic Autoencoder for Zero-Shot Learning (2017, methodology): “Inspired by humans’ ZSL ability, there is a recent surge of interest in machine learning based ZSL for scaling up visual recognition to unseen object classes without the need for additional data collection [66, 48, 19, 49, 2, 13, 58, 1, 54, 22, 23, 40, 11].”
Hierarchical Novelty Detection for Visual Object Recognition (2018, background): “, attributes [1, 18, 27] or text description [3, 9, 10, 23, 26, 30] of the objects.”
Dynamic Word Embeddings via Skip-Gram Filtering (2017, methodology): “Pre-trained word embeddings are useful for various supervised tasks, including sentiment analysis (Socher et al., 2013b), semantic parsing (Socher et al., 2013a), and computer vision (Fu & Sigal, 2016).”
MSplit LBI: Realizing Feature Selection and Dense Estimation Simultaneously in Few-shot and Zero-shot Learning (2018, methodology): “Our embedding model is compared against the state-of-the-art methods, including DAP/IAP (Lampert et al. (2014)), SJE (Akata et al. (2015)), SC_struct (Changpinyo et al. (2016)), LatEm (Xian et al. (2016)), LEESD (Ding et al. (2017)), SS-Voc (Fu and Sigal (2016)), DCL (Guo et al. (2017)), JLSE (Zhang”