Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs
By X. Wang, Yufei Ye, Abhinav Kumar Gupta. 2018
640 citations recorded by the source corpus.
We consider the problem of zero-shot recognition: learning a visual classifier for a category with zero training examples, just using the word embedding of the category and its relationship to other categories, which visual data are provided. The key to dealing with the unfamiliar or novel category is to transfer knowledge obtained from familiar classes to describe the unfamiliar class. In this paper, we build upon the recently introduced Graph Convolutional Network (GCN) and propose an approach that uses both semantic embeddings and the categorical relationships to predict the classifiers. Given a learned knowledge graph (KG), our approach takes as input semantic embeddings for each node (representing visual category). After a series of graph convolutions, we predict the visual classifier for each category. During training, the visual classifiers for a few categories are given to learn the GCN parameters. At test time, these filters are used to predict the visual classifiers of unseen categories. We show that our approach is robust to noise in the KG. More importantly, our approach provides significant improvement in performance compared to the current state-of-the-art results (from 2 ~ 3% on some metrics to whopping 20% on a few).
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Infectious Diseases, Representation & Self-Supervised Learning, Knowledge Graphs & Representation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Trevor Darrell
- Percy Liang
- Sergey Levine
- Yoshua Bengio
- Dacheng Tao
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Bernt Schiele
- Jitendra Malik
- Yi Yang
In Infectious Diseases
- Ping Hu
- Jun Shu
- Jacqueline C. K. Lam
- Stan Sclaroff
- Deyu Meng
- Victor O. K. Li
- Kate Saenko
- Yingruo Fan
In Representation & Self-Supervised Learning
- Trevor Darrell
- Fei Sha
- Boqing Gong
- Kate Saenko
- Zeynep Akata
- Gabriella Csurka
- Mingsheng Long
- Bernt Schiele
In Knowledge Graphs & Representation
- Ping Hu
- Jun Shu
- Jacqueline C. K. Lam
- Stan Sclaroff
- Deyu Meng
- Victor O. K. Li
- Kate Saenko
- Yingruo Fan
Citations in the corpus
- Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “In this sense, our work is also related to conventional zero-shot classification [9, 28, 45, 62, 65, 66].”
- Simplifying Graph Convolutional Networks (2019, methodology): “Following (Wang et al., 2018), we only apply dropout to the output of SGC. Due to the slow evaluation of this task, we do not tune the dropout rate or other hyperparameters.”
- Open-vocabulary Object Detection via Vision and Language Knowledge Distillation (2021, background): “Wang et al. (2018) distills information from both word embeddings and knowledge graphs.”
- Conditional Prompt Learning for Vision-Language Models (2022, background): “The most common approach to ZSL is to learn a semantic space based on auxiliary information such as attributes [23] or word embeddings [12, 52].”
- Generalizing from a Few Examples (2019, methodology): “is used in zero-shot learning frequently. They consider prior knowledge including attributes [1, 54], WordNet [1, 54], word embeddings [121, 128] , co-occurrence statistics [77], and knowledge graphs [129]. Recently, there are some works trying to adapt zero-shot learning methods to deal with FSL pr”
- LambdaNet: Probabilistic Type Inference using Graph Neural Networks (2020, methodology): “test time poses a challenge for traditional machine learning methods. For computer vision applications, solutions might involve looking at object attributes (Farhadi et al., 2017) or label similarity Wang et al. (2018); for natural language, similar techniques are applied to generalize across semant”
- Towards Discriminability and Diversity: Batch Nuclear-Norm Maximization Under Label Insufficient Situations (2020, methodology): “The graph structure is built according to the popular methods, Graph Convolutional Networks (GCN) [21, 49].”
- F-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning (2019, background): “[54] proposes to combine the semantic embedding and knowledge graph with graph convolutional network [24].”
- A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, methodology): “[182] proposed a multi-layer GCN to learn zero-shot classifiers using semantic embeddings of categories and categorical relationship.”
- KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, methodology): “Knowledge graphs have also found extensive use in visual classification and detection [53, 13], zero-shot classification [79] and image retrieval [32].”
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, background): “…can be mainly divided into two categories: ( i ) model-based methods focused on how to directly learn a model for unseen samples (Fu et al., 2017; Wang et al., 2018); and ( ii ) instance-based methods tried to obtain labeled instances for unseen tasks to improve model learning (Zhang et al.,…”
- Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (2019, methodology): “Automatic generation of medical image reports, as one of the key applications in the field, is gaining increasing research interest (Li et al. 2018; Wang et al. 2018b; Jing, Xie, and Xing 2018).”
- Zero-Shot Detection via Vision and Language Knowledge Distillation (2021, background): “[43] distills information from both word embeddings and knowledge graphs.”
- DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection (2022, background): “Under this setting, various works [5] try to find relationships between existing categories and unknown categories through pre-trained semantic/text features [46, 37, 40, 4, 13] knowledge graphs [43, 19, 51, 49] and so on.”
- Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation (2021, methodology): “GCNZ[40] and DGPZ[22] use a GCN to propagate knowledge into clas-sifiers of unseen classes, while using CNN and word embeddings to encode image and label features.”
- Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution (2020, methodology): “Due to the advantages in extracting the discriminative feature representations, GCNNs have been widely used in image classification (Wang, Ye, and Gupta, 2018), semantic segmentation (Liang et al., 2017), relational reasoning (Battaglia et al., 2016), etc.”