Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs

By X. Wang, Yufei Ye, Abhinav Kumar Gupta. 2018

640 citations recorded by the source corpus.

We consider the problem of zero-shot recognition: learning a visual classifier for a category with zero training examples, just using the word embedding of the category and its relationship to other categories, which visual data are provided. The key to dealing with the unfamiliar or novel category is to transfer knowledge obtained from familiar classes to describe the unfamiliar class. In this paper, we build upon the recently introduced Graph Convolutional Network (GCN) and propose an approach that uses both semantic embeddings and the categorical relationships to predict the classifiers. Given a learned knowledge graph (KG), our approach takes as input semantic embeddings for each node (representing visual category). After a series of graph convolutions, we predict the visual classifier for each category. During training, the visual classifiers for a few categories are given to learn the GCN parameters. At test time, these filters are used to predict the visual classifiers of unseen categories. We show that our approach is robust to noise in the KG. More importantly, our approach provides significant improvement in performance compared to the current state-of-the-art results (from 2 ~ 3% on some metrics to whopping 20% on a few).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Infectious Diseases, Representation & Self-Supervised Learning, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Trevor Darrell
  4. Percy Liang
  5. Sergey Levine
  6. Yoshua Bengio
  7. Dacheng Tao
  8. Quoc V. Le

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Bernt Schiele
  7. Jitendra Malik
  8. Yi Yang

In Infectious Diseases

  1. Ping Hu
  2. Jun Shu
  3. Jacqueline C. K. Lam
  4. Stan Sclaroff
  5. Deyu Meng
  6. Victor O. K. Li
  7. Kate Saenko
  8. Yingruo Fan

In Representation & Self-Supervised Learning

  1. Trevor Darrell
  2. Fei Sha
  3. Boqing Gong
  4. Kate Saenko
  5. Zeynep Akata
  6. Gabriella Csurka
  7. Mingsheng Long
  8. Bernt Schiele

In Knowledge Graphs & Representation

  1. Ping Hu
  2. Jun Shu
  3. Jacqueline C. K. Lam
  4. Stan Sclaroff
  5. Deyu Meng
  6. Victor O. K. Li
  7. Kate Saenko
  8. Yingruo Fan

Citations in the corpus