Towards Open World Object Detection

By K. J. Joseph, Salman Hameed Khan, Fahad Shahbaz Khan, Vineeth Nallure Balasubramanian. 2021

621 citations recorded by the source corpus.

Humans have a natural instinct to identify unknown object instances in their environments. The intrinsic curiosity about these unknown instances aids in learning about them, when the corresponding knowledge is eventually available. This motivates us to propose a novel computer vision problem called: ‘Open World Object Detection’, where a model is tasked to: 1) identify objects that have not been introduced to it as ‘unknown’, without explicit supervision to do so, and 2) incrementally learn these identified unknown categories without forgetting previously learned classes, when the corresponding labels are progressively received. We formulate the problem, introduce a strong evaluation protocol and provide a novel solution, which we call ORE: Open World Object Detector, based on contrastive clustering and energy based unknown identification. Our experimental evaluation and ablation studies analyse the efficacy of ORE in achieving Open World objectives. As an interesting by-product, we find that identifying and characterising unknown instances helps to reduce confusion in an incremental object detection setting, where we achieve state-of-the-art performance, with no extra methodological effort. We hope that our work will attract further research into this newly identified, yet crucial research direction.1

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Grounded Language-Image Pre-training (2021, background): “A line of work identifies building a openworld object proposal module that could propose any novel object at test time as the key challenge [22, 25, 60, 74, 75]; GLIP offers a new perspective: the model does not need to propose every possible novel objects from an open set; rather it only needs to p”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, neutral): “Open-world setup [5, 43] has been recently (re)-introduced into various problems in computer vision such as recognition [30, 38], tracking [36], detection [25, 27, 53], and segmentation [54].”
Learning Open-World Object Proposals without Learning to Classify (2021, background): “Our goal is to enhance learning-based proposals for promising applications including open-world detection [8] / segmentation [9], robot grasping [10] and egocentric video understanding [11].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “According to the provided X t and Y t in each D t , continual learning is further extended to zero-shot [23], few-shot [24], semi-supervised [25], open-world [26] and un-/self-supervised [27], [28] scenarios.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Since the data of the previous tasks are inaccessible, these methods typically enforce distillation objectives on the current data [32, 12], data from an exemplar memory [19, 13, 42, 7, 2, 27, 23], external data [30] or synthetic data [59].”
Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022, background): “the latest work, [15] introduced the concept of incremental learning when defining the problems of Open World Object Detection (OWOD).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Pellegrini et al. [104] stored activation volumes of previous data instances obtained from the intermediate layers, leading to much faster computation speed.”
Incremental Object Detection via Meta-Learning (2020, neutral): “[6], Faster ILOD [25] and ORE [24] .”
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation (2021, background): “L 1 loss B 1 B 2 ... 2021) incrementally learn identiﬁed unknown categories.”
Class-Incremental Learning: A Survey (2023, methodology): “…in classification into other tasks, e.g. , semantic segmentation [193], [238], [239], [240], [241], object detection [137], [242], [243], [244], [245], [246], [247], video understanding [91], [161], medical surgery [248], [249], language model [250], [251], [252], [253], language-vision…”
OW-DETR: Open-world Detection Transformer (2021, methodology): “Within the OWOD paradigm [15], at each training episode, a model learns to detect a given set of known objects while simultaneously”
Opening up Open World Tracking (2021, background): “[33] propose an extension to Faster R-CNN [66] for distinguishing known/unknown classes by adding a contrastive objective, that maximizes the margin between known and unknown objects in feature space.”
Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, background): “classification [49], [50], [51] and object detection [52], [53].”
Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection (2023, methodology): “Our improvements over the methods storing exemplars [21,27,28] confirm the importance of the box replay for IOD.”
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism (2022, positive): “Another line of researches [23, 34] on continual detection store a small number of samples(exemplars) for reviewing previous knowledge.”
VOS: Learning What You Don't Know by Virtual Outlier Synthesis (2022, methodology): “Moreover, Joseph et al. (2021) used the negative proposals as unknown samples for model regularization, which is suboptimal as we show in Table 2.”
Deep Class-Incremental Learning: A Survey (2023, neutral): “, semantic segmentation [164], [165], [166], [167], [174], object detection [175], [176], [177], [178], [179], [180], [181], video understanding [95], [133], medical surgery [182], [183], language model [184], [185], [186], [187], language-vision model [188], [189], etc.”
Spacing Loss for Discovering Novel Categories (2022, background): “Availability of large amount of annotated data has fueled unprecedented success of deep learning in various machine learning tasks [5, 9, 18, 30, 36, 37].”
Continual Detection Transformer for Incremental Object Detection (2023, background): “[19] proposes instead to store a set of exemplars and fine-tune the model on the exemplars after each incremental step.”