Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Concepts

By Shafin Rahman, Salman Hameed Khan, Fatih Murat Porikli. 2018

168 citations recorded by the source corpus.

Current Zero-Shot Learning (ZSL) approaches are restricted to recognition of a single dominant unseen object category in a test image. We hypothesize that this setting is ill-suited for real-world applications where unseen objects appear only as a part of a complex scene, warranting both the `recognition' and `localization' of an unseen category. To address this limitation, we introduce a new \emph{`Zero-Shot Detection'} (ZSD) problem setting, which aims at simultaneously recognizing and locating object instances belonging to novel categories without any training examples. We also propose a new experimental protocol for ZSD based on the highly challenging ILSVRC dataset, adhering to practical issues, e.g., the rarity of unseen objects. To the best of our knowledge, this is the first end-to-end deep network for ZSD that jointly models the interplay between visual and semantic domain information. To overcome the noise in the automatically derived semantic descriptions, we utilize the concept of meta-classes to design an original loss function that achieves synergy between max-margin class separation and semantic space clustering. Furthermore, we present a baseline approach extended from recognition to detection setting. Our extensive experiments show significant performance boost over the baseline on the imperative yet difficult ZSD problem.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

Learning Open-World Object Proposals without Learning to Classify (2021, background): “This line of research is also related to few-shot [10, 31, 67, 68, 71] and zero-shot detection [6, 20, 43, 70] methods, which attempt to detect novel classes given only a few samples or class descriptions.”
Incremental Few-Shot Object Detection (2020, neutral): “There are other techniques that aim to minimise the amount of data labelling for object detection such as weakly supervised learning [4, 10] and zero-shot learning [58, 37, 2].”
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation (2021, methodology): “Similar approaches have been used in prior detection works (Bansal et al., 2018; Rahman et al., 2018; Zareian et al., 2021).”
Few-Shot Object Detection via Feature Reweighting (2018, background): “Zeroshot object detection [1, 27, 42] aims to detect previously unseen object categories, thus usually requires external information such as relations between classes.”
Zero-Shot Object Detection (2018, background): “Comparison with recent works on ZSD : After completion of this work, we found two parallel works by Zhu et al. [61] and Rahman et al. [42] that target a similar problem.”
Don’t Even Look Once: Synthesizing Features for Zero-Shot Detection (2019, methodology): “[6] tabulates F-MNIST with clear black backgrounds, and [31] ImageNet with only single object/image, both of which are not informative from a detection perspective.”
Zero-Shot Semantic Segmentation (2019, background): “Very recently [4, 14, 34, 48] attacked object detection: unseen objects are detected while box annotation is only available for seen classes.”
RepMet: Representative-based metric learning for classification and one-shot object detection (2018, background): “Nevertheless, interesting recent works have demonstrated some preliminary results for the zero-shot object detection case [1, 22].”
Zero-Shot Object Detection: Joint Recognition and Localization of Novel Concepts (2020, background): “A preliminary version of this work appeared in [44].”
RepMet: Representative-Based Metric Learning for Classification and Few-Shot Object Detection (2018, background): “Recently, several interesting papers demonstrated preliminary results for the zero-shot object detection case [1, 23] and for the few-shot transfer learning [5] scenario.”
Probing Text Models for Common Ground with Visual Representations (2020, background): “Zero-shot detection Recent work attempts to build object detectors that generalize to previously unseen object categories, through conditioning the category predictions with word embeddings of the class (Rahman et al., 2018; Demirel et al., 2018), visual attributes (Demirel et al.”
Transductive Learning for Zero-Shot Object Detection (2019, background): “Existing efforts [1, 3, 24, 38, 22] to address the ZSD problem explore”
Meta-Learning to Detect Rare Objects (2019, background): “There are other settings of detection with limited supervision, such as weakly-supervised detection [59, 5, 11], semi-supervised detection [69, 39, 12], and zero-shot detection [2, 41, 73], which are different from ours.”
Zero Shot Detection (2018, methodology): “Last but not the least, this paper adopts a substantially more thorough evaluation setting with multiple splits of both ZSD and GZSD experiments, whereas both [38] and [38] are mainly evaluated on the ZSD test setting (only unseen classes) with only one split.”
One-Shot Instance Segmentation (2018, methodology): “Similarly our task shares similarities with zero-shot object detection [3, 42, 11, 69], but with the crucial difference that in zero-shot detection the reference category is defined by a textual description instead of an image.”