Multi-cue Zero-Shot Learning with Strong Supervision

By Zeynep Akata, Mateusz Malinowski, Mario Fritz, Bernt Schiele. 2016

152 citations recorded by the source corpus.

Scaling up visual category recognition to large numbers of classes remains challenging. A promising research direction is zero-shot learning, which does not require any training data to recognize new classes, but rather relies on some form of auxiliary information describing the new classes. Ultimately, this may allow to use textbook knowledge that humans employ to learn about new classes by transferring knowledge from classes they know well. The most successful zero-shot learning approaches currently require a particular type of auxiliary information - namely attribute annotations performed by humans - that is not readily available for most classes. Our goal is to circumvent this bottleneck by substituting such annotations by extracting multiple pieces of information from multiple unstructured text sources readily available on the web. To compensate for the weaker form of auxiliary information, we incorporate stronger supervision in the form of semantic part annotations on the classes from which we transfer knowledge. We achieve our goal by a joint embedding framework that maps multiple text parts as well as multiple semantic parts into a common space. Our results consistently and significantly improve on the state-of-the-art in zero-short recognition and retrieval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Learning to Reconstruct Shapes from Unseen Classes (2018, background): “More recently, many researchers have begun to study zero- or few-shot recognition with deep networks [Akata et al., 2016, Antol et al., 2014, Hariharan and Girshick, 2017, Wang et al., 2017, Wang and Hebert, 2016].”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly (2017, methodology): “Furthermore, hybrid models [14], [38], [39], [40] such as [39] jointly embeds multiple text representations and multiple visual parts to ground attributes on different”
Zero-Shot Learning — The Good, the Bad and the Ugly (2017, neutral): “compatibility learning frameworks [3, 4, 11, 15, 27, 32, 34, 39, 32, 12, 29, 1, 6, 24, 13, 21].”
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation (2021, background): “…direction, researchers have also explored class hierarchy, class similarity, and object parts as discriminative features to aid the knowledge transfer (Rohrbach et al., 2011; Akata et al., 2016; Zhao et al., 2017; Elhoseiny et al., 2017; Ji et al., 2018; Cacheux et al., 2019; Xie et al., 2020).”
Feature Generating Networks for Zero-Shot Learning (2017, background): “[49, 31, 8] learn unseen classes as a mixture of seen class proportions, and [2, 3, 14, 42, 45, 40, 16, 33, 1, 6, 17, 23] learn a compatibility between images and classes.”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, background): “…techniques (Krishnamurthy and Kollar, 2013; Matuszek et al, 2012), there is a recent trend of machine learning-based approaches (Karpathy and Fei-Fei, 2015; Karpathy et al, 2014; Akata et al, 2016; Kong et al, 2014; Hu et al, 2016; Rohrbach et al, 2015a; Mao et al, 2016) to find the associations.”
Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts (2018, methodology): “Note that, we do not use per image part annotation (like [1]) and descriptions (like [51]) to enrich semantic embedding.”
Normalization Matters in Zero-Shot Learning (2020, background): “References [1] Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele.”
Zero-Shot Learning with Generative Latent Prototype Model (2017, background): “…unseen categories being disjoint, the key in the general methodology of ZSL is to establish the inter-class connections via intermediate semantic representations, either manually deﬁned by human experts annotated attributes [5]–[9], or automatically extracted from auxiliary text sources [10]–[15].”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, background): “Mapping both image and semantic features into another common intermediate space is another direction that ZSL approaches adapt [66, 16, 67, 1, 28].”
Zero-Shot Detection via Vision and Language Knowledge Distillation (2021, background): “Researchers have also explored class hierarchy, class similarity, and object parts as discriminative features to aid the knowledge transfer [38, 1, 8, 23, 4, 44].”
Zero-Shot Object Detection: Joint Recognition and Localization of Novel Concepts (2020, background): “Since its inception, the main focus of zero-shot learning research has been object classification (Akata et al. 2016; Changpinyo et al. 2016; Frome et al. 2013; Kodirov et al. 2015;Lampert et al. 2014;LeiBa et al. 2015;MaximeBucher and Jurie 2016; Morgado and Vasconcelos 2017; RomeraParedes and Torr”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “[57] proposed to extract semantic representations from multiple textual”
Beyond Instance-Level Image Retrieval: Leveraging Captions to Learn a Global Visual Representation for Semantic Retrieval (2017, background): “Many tasks require to jointly leverage images and natural text, such as zeroshot learning [4, 10], language generation [67, 35], multimedia retrieval [2, 3], image captioning [62, 16], and VQA [59, 45, 6].”
Region Graph Embedding Network for Zero-Shot Learning (2020, methodology): “Initial works [11, 1, 64] utilized part annotations to discover discriminative part features for tackling ﬁne-grained ZSL.”
A Survey of Zero-Shot Learning (2019, background): “In image object recognition, in addition to features about the whole image, features about different parts of the objects can also be considered [1, 8, 32].”
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts (2017, methodology): “Note that the conventional ZSL setting is SCS-split used in [1, 32, 36, 3].”
Generating Visual Representations for Zero-Shot Classification (2017, methodology): “id over-ﬁtting, we used dropout [35] at every layer (probability of drop of 0.2 for the inputs layers and of 0.5 for the hidden layers). Input data (both image features and w2c vectors) are scaled to [0,1] by applying an afﬁne transformation. With the TensorFlow framework [13] running on a Nvidia Ti”