Towards Open World Recognition

By Abhijit Bendale, Terrance E. Boult. 2014

653 citations recorded by the source corpus.

With the of advent rich classification models and high computational power visual recognition systems have found many operational applications. Recognition in the real world poses multiple challenges that are not apparent in controlled lab environments. The datasets are dynamic and novel categories must be continuously detected and then added. At prediction time, a trained system has to deal with myriad unseen categories. Operational systems require minimal downtime, even to learn. To handle these operational issues, we present the problem of Open World Recognition and formally define it. We prove that thresholding sums of monotonically decreasing functions of distances in linearly transformed feature space can balance “open space risk” and empirical risk. Our theory extends existing algorithms for open world recognition. We present a protocol for evaluation of open world recognition systems. We present the Nearest Non-Outlier (NNO) algorithm that evolves model efficiently, adding object categories incrementally while detecting outliers and managing open space risk. We perform experiments on the ImageNet dataset with 1.2M+ images to validate the effectiveness of our method on large scale visual recognition tasks. NNO consistently yields superior results on open world recognition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Learning Visual N-Grams from Web Data (2016, background): “The assumptions that are implicit in such benchmarks are at odds with many real-world applications of imagerecognition systems, which often need to be deployed in an open-world setting [3].”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, neutral): “Open-world setup [5, 43] has been recently (re)-introduced into various problems in computer vision such as recognition [30, 38], tracking [36], detection [25, 27, 53], and segmentation [54].”
Deep Variational Information Bottleneck (2017, methodology): “…real images; using richer parametric marginal approximations, rather than assuming r ( z ) = N (0 , I ) ; exploring the connections to differential privacy (see e.g., Wang et al. (2016a); Cuff & Yu (2016)); and investigating open universe classiﬁcation problems (see e.g., Bendale & Boult (2015)).”
Towards Open Set Deep Networks (2015, methodology): “To recognize outliers using AVs, we adapt the concepts of Nearest Class Mean [29, 12] or Nearest Non-Outlier [1] and apply them per class within the activation vector, as a first approximation.”
iCaRL: Incremental Classifier and Representation Learning (2016, background): “[37], as well as Bendale and Boult [2] aimed at the related but distinct problem of Open Set Recognition in which test examples might come from other classes than the training examples seen so far.”
Few-Shot Continual Active Learning by a Robot (2022, background): “Another related field to active learning is open-set recognition (or out of the distribution detection (OOD)) [37, 38, 38, 39], in which a model that is trained on a set of tasks/classes might face samples belonging previously unknown classes in the test set.”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “In testing, a distance function over the nearest exemplar/class mean is used for classiﬁcation (Rebufﬁ et al. 2017; Lee et al. 2018; Javed and Shafait 2018; Bendale and Boult 2015).”
Learngene: From Open-World to Your Learning Task (2021, background): “…cannot resolve incrementally learning the unknown/unseen classes (Fei, Wang, and Liu 2016; Xu et al. 2019) without retraining from scratch and without catastrophic forgetting (French 1999), which is exactly what open-world learning (Bendale and Boult 2015; Shu, Xu, and Liu 2018) wants to solve.”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “In testing, a distance function over nearest exemplar/class mean is used for classiﬁcation (Re-bufﬁ, Kolesnikov, and Lampert 2017; Lee et al. 2018; Javed and Shafait 2018; Bendale and Boult 2015).”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “A natural downstream setting that our work facilitates is open world learning [7] or automatic class discovery [63, 75], where an agent must identify samples outside of its training distribution as unknown and then learn them.”
Towards Open World Object Detection (2021, background): “Open World Classiﬁcation: [3] ﬁrst proposed the open world setting for image recognition.”
The limits and potentials of deep learning for robotics (2018, background): “In these so called open-set conditions [11], [101], it is crucial to identify the unknowns: The perception system must not assign high-conﬁdence scores to unknown objects or falsely recognize them as one of the known classes.”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “Open world learning [84], [85] deals with the problem of detecting new classes at test time, hence avoiding wrong assignments to known classes.”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “Correspondingly, works that provide attempts at addressing open set recognition aim to find the respective boundaries between known and unknown spaces [39], [40], [127], [128], [131], [132], [133].”
Learning Placeholders for Open-Set Recognition (2021, background): “However, this closed-world assumption comes to an end when the test set includes unseen categories [16, 2, 42, 35].”
BranchyNet: Fast inference via early exiting from deep neural networks (2016, methodology): “For future work, we plan to explore Meta-Recognition algorithms, such as OpenMax, to automatically adapt T to new test samples.”
Learning on the Job: Online Lifelong and Continual Learning (2020, background): “Several researchers have studied this problem, e.g., Scheirer et al. (2013), Bendale and Boult, 2015; Fei and Liu (2016), De Rosa et al (2016), and Xu et al. (2019).”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, background): “Openworld problems have been studied in the context of recognition [3, 25]: given a closed-world training dataset, how to actively identify new object categories? To handle novel objects, previous works explicitly differentiate unknown from known, such as by spotting outliers in the embedding space.”
Recent Advances in Open Set Recognition: A Survey (2018, methodology): “To jointly consider the OSR and CIL tasks, Bendale and Boult [72] expanded the existing open set recognition (Definition 2) to the open world recognition (OWR), where a recognition system should perform four tasks: detecting UUCs, choosing which samples to label for addition to the model, labelling ”