Improved Visual-Semantic Alignment for Zero-Shot Object Detection

By Shafin Rahman, Salman Hameed Khan, Nick Barnes. 2020

124 citations recorded by the source corpus.

Zero-shot object detection is an emerging research topic that aims to recognize and localize previously ‘unseen’ objects. This setting gives rise to several unique challenges, e.g., highly imbalanced positive vs. negative instance ratio, proper alignment between visual and semantic concepts and the ambiguity between background and unseen classes. Here, we propose an end-to-end deep learning framework underpinned by a novel loss function that handles class-imbalance and seeks to properly align the visual and semantic cues for improved zero-shot learning. We call our objective the ‘Polarity loss’ because it explicitly maximizes the gap between positive and negative predictions. Such a margin maximizing formulation is not only important for visual-semantic alignment but it also resolves the ambiguity between background and unseen objects. Further, the semantic representations of objects are noisy, thus complicating the alignment between visual and semantic domains. To this end, we perform metric learning using a ‘Semantic vocabulary’ of related concepts that refines the noisy semantic embeddings and establishes a better synergy between visual and semantic domains. Our approach is inspired by the embodiment theories in cognitive science, that claim human semantic understanding to be grounded in past experiences (seen objects), related linguistic concepts (word vocabulary) and the visual perception (seen/unseen object images). Our extensive results on MS-COCO and Pascal VOC datasets show significant improvements over state of the art.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Multimodal Learning

Citations in the corpus

RegionCLIP: Region-based Language-Image Pretraining (2021, background): “Zeroshot object detection aims at detecting novel object classes that are not seen during detector training [2, 18, 38, 39, 60, 65].”
Grounded Language-Image Pre-training (2021, methodology): “Our setting differs from zero-shot detection [1,12,14,46,47,68], where some categories are de-ﬁned as unseen/rare and not present in the training set.”
Open-Vocabulary Object Detection Using Captions (2020, methodology): “…is to improve the Nevertheless, ZSD methods are still far from practical performance, due to their unnecessarily harsh constraint, i.e ., not having any example of unseen objects, and having to guess how they look like solely based on their word embed-dings [3, 30, 46] or textual descriptions [24].”
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels (2021, methodology): “Recent methods encourage the visual-semantic alignment for novel objects by different strategies such as synthesizing visual representations of novel classes [35,36] or utilizing existing object names semantically similar to their names [22].”
Zero-Shot Object Detection: Joint Recognition and Localization of Novel Concepts (2020, background): “Recently, a new polarity loss for zero-shot detection has been proposed in [39,41] and achieved signiﬁcant performance boost on MSCOCO-2014 and Pascal VOC-07 datasets.”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, methodology): “Many methods address this by aligning the image region features to category word embeddings (Bansal et al., 2018; Rahman et al., 2020; Demirel et al., 2018; Zheng et al., 2020), or synthesizing visual features with a generative model (Hayat et al.”