Zero-Shot Object Detection

By Ankan Bansal, Karan Sikka, Gaurav Sharma, Ramalingam Chellappa, Ajay Divakaran. 2018

418 citations recorded by the source corpus.

We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training. We work with a challenging set of object classes, not restricting ourselves to similar and/or fine-grained categories cf. prior works on zero-shot classification. We follow a principled approach by first adapting visual-semantic embeddings for ZSD. We then discuss the problems associated with selecting a background class and motivate two background-aware approaches for learning robust detectors. One of these models uses a fixed background class and the other is based on iterative latent assignments. We also outline the challenge associated with using a limited number of training classes and propose a solution based on dense sampling of the semantic label space using auxiliary data with a large number of categories. We propose novel splits of two standard detection datasets - MSCOCO and VisualGenome and discuss extensive empirical results to highlight the benefits of the proposed methods. We provide useful insights into the algorithm and conclude by posing some open questions to encourage further research.

Subfields: Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Florence: A New Foundation Model for Computer Vision (2021, methodology): “, 2021a) (on COCO) fine tuning results on full-set or 5-shot respectively and zero-shot detection baseline model ZSD (Bansal et al., 2018).”
RegionCLIP: Region-based Language-Image Pretraining (2021, methodology): “5 - - Cls-IncRN [49] ImageNet IncRNv2 SB [2] IncRNv2 0.”
Grounded Language-Image Pre-training (2021, methodology): “1, the model can be viewed as a strong classical zero-shot detection model [1], relying purely on the language encoder to generalize to new concepts.”
Learning Open-World Object Proposals without Learning to Classify (2021, background): “This line of research is also related to few-shot [10, 31, 67, 68, 71] and zero-shot detection [6, 20, 43, 70] methods, which attempt to detect novel classes given only a few samples or class descriptions.”
Incremental Few-Shot Object Detection (2020, neutral): “There are other techniques that aim to minimise the amount of data labelling for object detection such as weakly supervised learning [4, 10] and zero-shot learning [58, 37, 2].”
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation (2021, methodology): “COCO: Bansal et al. (2018) divide COCO-2017 (Lin et al., 2014) into 48 base categories and 17 novel categories, removing 15 categories without a synset in the WordNet hierarchy.”
Open-Vocabulary Object Detection Using Captions (2020, methodology): “Nevertheless, this approach is prone to overfitting, as projecting to a small number of the embedding space (base class embeddings) is an under-determined problem [3].”
Few-Shot Object Detection via Feature Reweighting (2018, background): “Zeroshot object detection [1, 27, 42] aims to detect previously unseen object categories, thus usually requires external information such as relations between classes.”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “beddings to create open-vocabulary detectors [2].”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Many recent approaches have started to study the open-set and zero-shot settings [15,2,84,98,20].”
Zero-Shot Semantic Segmentation (2019, background): “Very recently [4, 14, 35, 51] attacked object detection: unseen objects are detected while box annotation is only available for seen classes.”
Consistent Structural Relation Learning for Zero-Shot Segmentation (2020, methodology): “Despite on the traditional image classiﬁcation task, ZSL has been applied to predict novel action in videos [22, 23], detect unseen objects [24, 25], and recently, to segment pixel-wise unseen categories [26, 1].”
Opening up Open World Tracking (2021, methodology): “This recall-based evaluation is inspired by prior work for evaluating tasks in the open-world such as zero-shot object detection or LiDAR instance segmentation [4, 84].”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, background): “…in any environment with unstructured language input is challenging; however, recent advances in open-vocabulary models for image classiﬁcation [35, 58, 61], object detection [4, 21, 27, 36, 43, 47, 49, 62, 88], and semantic segmentation [3, 6, 15, 33, 36, 37, 86] present a promising foundation.”
Improved Visual-Semantic Alignment for Zero-Shot Object Detection (2020, methodology): “…2018b; Zhu et al. 2018), ( e ) inability to predict bounding boxes spe-ciﬁc to unseen objects (Rahman, Khan, and Porikli 2018b; Bansal et al. 2018), ( f ) reliance on pre-trained weights tuned on datasets containing unseen objects (Bansal et al. 2018; Demirel, Cinbis, and Ikizler-Cinbis 2018).”
Zero Shot Detection (2018, methodology): “Two other groups have concurrently worked on ZSD [37], [38] and appeared around the time a preliminary version of this paper was posted on arxiv.”
Residual Tuning: Toward Novel Category Discovery Without Labels (2022, neutral): “2) As aforementioned, we note that NCD is different from ZSD [10] where unseen classes are used only at the test phase and they must be auxiliary semantic or label embeddings are needed for all the classes.”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “Such a noisy dataset will inevitably result in a poor performance of the trained object detector [3], which further hinders the performance of predicate detection.”