Multi-label Zero-Shot Learning with Structured Knowledge Graphs

By Chung-Wei Lee, Wei Fang, Chih-Kuan Yeh, Yu-Chiang Frank Wang. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

307 citations recorded by the source corpus.

In this paper, we propose a novel deep learning architecture for multi-label zero-shot learning (ML-ZSL), which is able to predict multiple unseen class labels for each input instance. Inspired by the way humans utilize semantic knowledge between objects of interests, we propose a framework that incorporates knowledge graphs for describing the relationships between multiple labels. Our model learns an information propagation mechanism from the semantic label space, which can be applied to model the interdependencies between seen and unseen class labels. With such investigation of structured knowledge graphs for visual reasoning, we show that our model can be applied for solving multi-label classification and ML-ZSL tasks. Compared to state-of-the-art approaches, comparable or improved performances can be achieved by our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/17a8dd804ff254662cea2450c25dfa0bb7adceb2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Zero-Shot Learning (2019, methodology): “When learning the classifier, existing methods either treat each class label individually [14, 99, 100, 168] or exploit multilabel correlations [44, 84, 107, 126, 172].”
Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects (2019, background): “transductive ZSL [15, 20], multi-label ZSL [18, 31], zero-shot detection [29] etc.”
Recent Advances in Open Set Recognition: A Survey (2018, methodology): “It is also worthy mentioning that the datasets NUS-Wide and MS COCO have been used for studying multi-label zeroshot learning [161], which are suitable to the study of multilabel OSR problem as well.”
Concept Generalization in Visual Representation Learning (2020, unknown): “A handful of methods leverage existing ontologies, for instance the WordNet graph [41] is used as a reference for splitting [12, 17, 33, 73] or domainspecific ontologies are used to test cross-domain generalization [21, 63].”
General Multi-label Image Classification with Transformers (2020, background): “Multi-label image classi-ﬁcation is a visual recognition task that aims to predict a set of labels corresponding to objects, attributes, or actions given an input image [15, 46, 48, 50, 6, 32, 9].”
Transductive Zero-Shot Learning for 3D Point Cloud Classification (2019, methodology): “Zero-Shot Learning: For the ZSL task, there has been significant progress, including on image recognition [35, 65, 2, 4, 29, 21, 57], multi-label ZSL [22, 34], and zero-shot detection [36].”
A Review of Generalized Zero-Shot Learning Methods (2020, unknown): “multi-label [188], [189], multi-view (CT and x-ray ) [190]”
Multi-Label Image Recognition With Graph Convolutional Networks (2019, methodology): “In our experiments, following [36, 1, 15, 6], we use ResNet-101 [10] as the base model in experiments.”
Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (2020, background): “There’s also a popular trend to model label correlation with graph structure [21,4].”
Small Sample Learning in Big Data Era (2018, background): “Another work employed information propagation mechanism to reason the unseen labels is proposed by (Lee et al., 2018a).”
Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “Incorporating commonsense knowledge has been explored in various computer vision tasks such as object recognition [3,28,14], object detection [13], semantic segmentation [19], action recognition [9], visual relation detection [31], scene graph generation [2,34,7], and visual question answering [18,2”
Semantic Projection Network for Zero- and Few-Label Semantic Segmentation (2019, background): “As complementary tasks [24, 40] focuses on assigning multiple previously unseen labels to a single image, [22, 52, 53] on predicting novel actions in a video, and recently, [4] on unseen object detection.”
Knowledge-Embedded Routing Network for Scene Graph Generation (2019, methodology): “Lee et al. [15] further extended this method to multi-label zero-shot learning.”
The Emerging Trends of Multi-Label Learning (2020, methodology): “[36] advances label propagation mechanism in the semantic space, enabling the reasoning of the learned model for predicting unseen labels.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “For example, zero-shot [126], [132], [133], one-shot”
Zero-shot Learning of 3D Point Cloud Objects (2019, background): “In the 2D image domain, on the contrary, there are many works addressing the ZSL problem in the past few years [19, 32, 9, 3].”